META tags were the original equalizer against the intrusion of big business advertising. A small company could produce a 'tag' that provided a measure of free advertising without incurring cost. Growing companies operating search engines were all too happy to spider a site willing to give them free value. This free 'give and take' allowed the search engine owners to grow, and provided an opportunity for a small company doing business on the internet to advertise. Now, the main search engine companies have blossomed into multi-million dollar businesses with stockholders and Boards of Directors.
Investors are eager for the search engine companies to show a profit.
The questions Yahoo faces now, are not the same questions they faced two years ago. Mature search engine companies do not really need to spider every applicant. At this point in their corporate development, finding ways to generate revenue is a principal goal. That is what every company strives for. We may not like it but, the Internet is transforming into a self contained business entity.