Lazily load Ext::Builder classes

This tries to fix #784