Fix error when destroy is called before the field is focused

Fixes an error that occurs if destroy is called before the input has received focus.