Lee 's popularity ratings have tumbled to the low 20s , according to opinion polls , and his conservative Grand National Party suffered big losses in local by-elections Wednesday , its first electoral test since taking office .