A search engine penalty is an action taken against a website engaged in deceptive or black hat SEO practices. Google's definition of a "penalty" is some manual action taken against a site, contrasted to the automated consequence of an algorithm or quality filter.