In the notes, RFC 2616 seems to recommend against a HTTP client using its own time for If-Modified-Since:

```
      Note: If a client uses an arbitrary date in the If-Modified-Since
      header instead of a date taken from the Last-Modified header for
      the same request, the client should be aware of the fact that this
      date is interpreted in the server's understanding of time. The
      client should consider unsynchronized clocks and rounding problems
      due to the different encodings of time between the client and
      server. This includes the possibility of race conditions if the
      document has changed between the time it was first requested and
      the If-Modified-Since date of a subsequent request, and the
      possibility of clock-skew-related problems if the If-Modified-
      Since date is derived from the client's clock without correction
      to the server's clock. Corrections for different time bases
      between client and server are at best approximate due to network
      latency.
```
According to RFC every response SHOULD include a Date header, but servers without reliable time MUST NOT include them. So I think it's safest if we do not use If-Modified-Since with a server that's too broken to send a Date header.