Realized I was breaking spec by not sending down the same required headers with a 304 vs. a 200.  Specifically, the ETag and Last-Modified headers were missing from a 304 Not Modified response although required:

http://tools.ietf.org/html/draft-ietf-httpbis-p4-conditional-12#section-3.1