Ensure time adjustment widgets remain insensitive

If the calendar date was adjusted while all-day is selected, the
time adjustment widgets would become sensitive.  This prevents that.