If min and max are just the date, set date. If time, set time.

When using the directive as a date picker, just set the date.
When using the directive as a time picker, just set the time. 
Fall back to current functionality for other cases. 

Not sure if this is right for the project, but I'm pushing it out in case anyone is interested.