mozilla/browserid — pull request #2888 by lloyd (2012-12-20 04:59:30+00:00)
Title: fix cache poisioning introduced by borked production push

The fix associated with this email:

there was an event recently where we were running some number of our servers in an odd state. They had updated resources but were not restarted. Shane did a pro-active assessment of impact on this list, and subsequent to that we received a report of a broken dialog in issue 2863.

Skipping some parts and zooming ahead - the explanation from Shane is that we might see the screenshots in that issue if NEW html was served with OLD css.

In auditing etagify and cachify (the libraries that we built and use to generate good cache headers) - the only path I can see where this were be plausible is if *some* of the processes in a single data center were updated while others weren't. In this case we could get an html request routed to a server serving NEW resources with embedded "cachified" urls. If those cachified urls were then fetched from a server that was not updated with new resources, it could serve OLD data under the NEW cachified url -- Poisioning the cache for css/js resources.

Gene: Was it the case that some servers in a single data center were updated while others were not?

All of this aside, we can push a change that will invalidate browser caches for all of the resources that we serve, and I feel fairly confident that this will resolve any issues. To accomplish this we simply need to change combined css/js in a way that will alter minified output (cannot be whitespace only, which would be lost in minification).

Now the question is, should we? Given that we have 3 internal reproductions of the issue and at least one external, should we craft a tiny commit to alter our resources and guarantee that this does not adversely affect users that are not speaking up?

If we don't create and push this fix this week, then it seems like we just live with adverse affects.

For completeness, a user workaround is to force reload the dialog or clear their cache.