As a result, bandwidth and compute resources would be used more efficiently since indexes would hit websites only once. The PICS format is one possibility. Communication A web server is optimized towards serving a single document at a time but an indexer wants collections of documents to work with. Either we need to come up with a special URL to allow indexers easy access to collections or we need to introduce a new service geared to the needs of indexers. The current crop of NSF-funded digital library projects can have a large bearing on this discussion.
More work is needed to deal with simple things such as accented letters and varying character sets.
A document may exist in multiple language versions, all of which may exist under the same URL depending on the Accept-Language headers that are sent. Meta-data is required to describe the dimensions on which a document may vary. In summary, there needs to be more cooperation between indexers and document servers in order to make better use of scarce resources. Content providers have to provide more meta-data as servers and document publishing systems become more complex. Exposing this data to the world in a standard way will add tremendous value to the document collection and ultimately make information more accessible and useful.