iipc/openwayback — pull request #182 by scheylord (2014-10-30 16:22:29+00:00)
Title: Add a system to manage specific rules for some URL (spring configurable)

Hello everyone,

As we are discuting about the future of the OpenWayback project, the BnF would like to present you a small feature that could maybe interest some of you and could be add to the openwayback project.

This feature has been developed by Nicolas Giraud, the previous software engineer of the BnF web archive team. This feature provides a system to manage specific canonicalization rules on URL. For instance, we have a news website which have session id in URLs and we want to remove it during the search and replay process.

The URLs looks like :
http://www.ouestfrance-enligne.com/scripts/consult/pdf/PDF_frame.asp?in_ses_id=2047240875806&pdf=S2192220&date=09/09/2014&art_id=68159490&zoom=125,11,78

We define the rules in Spring :
```

```
- BnfUrlCanonicalizer simply extends AggressiveUrlCanonicalizer
- we have a processingRules property which is a list of canonicalization rules
- A CanonicalizationRule is a pattern, in our case it's a specific domain (ouestfrance-enligne.com), and a list of processors that will only process when some input URL matchs this pattern.
- the processors could be from several types, but all extends PatternBasedTextProcessor abstract class :
* UriStripper remove the specific pattern in an input URL
* UriTranscoder transcode the matching part of an URL from a encoding to an other

In this example, we want to remove the in_ses_id attribut, so we use a UriStripper processor with the corresponding pattern.
Then we also want to transcode an another attribute that cause trouble, from ISO-8859-1 to UTF-8 so we use the UriTranscoder with the right pattern.

In BnfUrlCanonicalizer, we override urlStringToKey method from AggressiveUrlCanonicalizer, adding :
```
for (CanonicalizationRule rule : getProcessingRules()) {
searchUrl = rule.processIfMatches(new CanonicalizationInput(searchUrl));
}
```
which will transform the URL in case of match.

We could give to the OpenWayback project all the classes needed to run this functionality.