The server won't return 304 if he didn't have some sort of header for caching in the request, no need to check it.
Also you may only send one of the two headers and receive only one. This is still a valid scenario.