Flexget/Flexget — pull request #91 by Toilal (2013-10-19 14:29:16+00:00)
Title: Add incremented variable to support paginated pages

html plugin is often used on search webpages that are paginated. Configuration to retrieve all required pages is fastidious and hard to maintain.

Here's the configuration i needed to use before (see page parameter in each url).

```
series:
inputs:
- html:
url: "www.somesite.com/search/?category=10&page=0"
title_from: title
links_re:
- "^www.somesite.com/torrents/[^/]+(?!/)[^/]+$"
- html:
url: "www.somesite.com/search/?category=10&page=1"
title_from: title
links_re:
- "^www.somesite.com/torrents/[^/]+(?!/)[^/]+$"
- html:
url: "www.somesite.com/search/?category=10&page=2"
title_from: title
links_re:
- "^www.somesite.com/torrents/[^/]+(?!/)[^/]+$"
- html:
url: "www.somesite.com/search/?category=10&page=3"
title_from: title
links_re:
- "^www.somesite.com/torrents/[^/]+(?!/)[^/]+$"
- html:
url: "www.somesite.com/search/?category=10&page=4"
title_from: title
links_re:
- "^www.somesite.com/torrents/[^/]+(?!/)[^/]+$"
- html:
url: "www.somesite.com/search/?category=10&page=5"
title_from: title
links_re:
- "^www.somesite.com/torrents/[^/]+(?!/)[^/]+$"
- html:
url: "www.somesite.com/search/?category=10&page=6"
title_from: title
links_re:
- "^www.somesite.com/torrents/[^/]+(?!/)[^/]+$"
- html:
url: "www.somesite.com/search/?category=10&page=7"
title_from: title
links_re:
- "^www.somesite.com/torrents/[^/]+(?!/)[^/]+$"
- html:
url: "www.somesite.com/search/?category=10&page=8"
title_from: title
links_re:
- "^www.somesite.com/torrents/[^/]+(?!/)[^/]+$"
- html:
url: "www.somesite.com/search/?category=10&page=9"
title_from: title
links_re:
- "^www.somesite.com/torrents/[^/]+(?!/)[^/]+$"
```

And here's the replacement
```
series:
html:
url: "www.somesite.com/search/?category=10&page={i}"
increment:
from: 0
to: 10
title_from: title
links_re:
- "^www.somesite.com/torrents/[^/]+(?!/)[^/]+$"
```

It'll have the same effect as the previous pasted configuration. The plugin will increment the value of variable "i", and then retrieve entries by performing the request. The code beetween {...} is evaluated like in "if_condition" plugin, so basic operations can be performed.

Here's another use case (step parameter, and multiple use of {...})
```
series:
html:
url: "www.somesite.com/search/?category=10&from={i}&to={i+49}"
increment:
from: 0
to: 10
step : 50
title_from: title
links_re:
- "^www.somesite.com/torrents/[^/]+(?!/)[^/]+$"
```

This configuration will grab entries from those URLs
```
www.somesite.com/search/?category=10&from=0&to=49
www.somesite.com/search/?category=10&from=50&to=99
www.somesite.com/search/?category=10&from=100&to=149
www.somesite.com/search/?category=10&from=150&to=199
www.somesite.com/search/?category=10&from=200&to=249
www.somesite.com/search/?category=10&from=250&to=299
www.somesite.com/search/?category=10&from=300&to=349
www.somesite.com/search/?category=10&from=350&to=399
www.somesite.com/search/?category=10&from=400&to=449
www.somesite.com/search/?category=10&from=450&to=499
```

Configuration can be as consise as this
```
series:
html:
url: "www.somesite.com/search/?category=10&from={i}&to={i+49}"
increment: True
title_from: title
links_re:
- "^www.somesite.com/torrents/[^/]+(?!/)[^/]+$"
```

Options
```from``` First value of the increment [integer, default 0]
```to``` If defined, plugin will stop when increment reach this value [integer, default None]
```name``` Name of the variable used in {...} evaluated blocks [integer, default "i"]
```step``` The value that will be added to the variable, for each iteration (Can be negative) [integer, default 1]
```stop_when_empty``` If True, plugin will stop when a page retrieves no entry [boolean, default True]
```entries_count``` If total entries count exceed this value, plugin will stop [integer, default 200]