Refuse to downcast from datetime to date or time