Q:

Why does HTML5 video with very large h.264 encoded mp4 (with +faststart, ie metadata at beginning), take ages to load? 

The video is rendered with ffmpeg with the "faststart" flag added meaning the metadata should be at the start of the file, and the server appears to be handling partial content requests correctly, so why does it need to have downloaded so much of the video before the player becomes enabled and can play the video?  I am testing it in Google Chrome. 
Once the player becomes enabled I can seek around to various points in the video pretty instantly and see the new partial content requests being made. 
Here is a link to the video: http://iptv.la1tv.co.uk/unibrass.mp4
Here is a jsbin with the video tag: https://jsbin.com/rahewidoru .  It takes a few minutes but does work after it loads. 
Any suggestions? 
I realise there are other methods like HLS and dash which use chunks, but I would like to know why it isn't working this way because I can't find anywhere that provides a reason as to why it doesn't work well. 

A:

So it looks like it is just how h.264 works.  The metadata contains pointers to all the keyframes in the file, and therefore as the duration of the file increases, so does the number of keyframes, and hence the size of metadata at the beginning. 
More information at http://www.adobe.com/devnet/video/articles/mp4_movie_atom.html
If anyone can provide a more in depth answer as to what the metadata in a h.264 encoded mp4 contains then let me know and I'll mark that as the answer, or just edit this.