Yea, I think a module should be the one deciding whether or not the web font loader attempts to load fonts. The downside is that some modules load external JavaScript before deciding on support.