Package: django-proxylist-for-grab
Django-ProxyList-For-Grab=========================.. image:: https://api.travis-ci.org/gotlium/django-proxylist.png?branch=master:alt: Build Status:target: https://travis-ci.org/gotlium/django-proxylist.. image:: https://coveralls.io/repos/gotlium/django-proxylist/badge.png?branch=master:target: https://coveralls.io/r/gotlium/django-proxylist?branch=master.. image:: https://pypip.in/v/django-proxylist-for-grab/badge.png:alt: Current version on PyPi:target: https://crate.io/packages/django-proxylist-for-grab/.. image:: https://pypip.in/d/django-proxylist-for-grab/badge.png:alt: Downloads from PyPi:target: https://crate.io/packages/django-proxylist-for-grab/This application is useful for keep an updated list of proxy servers, itcontains everything you need to make periodic checks to verify the propertiesof the proxies. Also you can periodically collect the proxy serverfrom the Internet, remove broken and slow proxies.Installing the package----------------------`django-proxylist-for-grab` can be easily installed using pip:.. code-block:: bash$ pip install django-proxylist-for-grabConfiguration-------------After that you need to include `django-proxylist-for-grab` into your*INSTALLED_APPS* list of your django settings file... code-block:: pythonINSTALLED_APPS = (...'proxylist',...)Add `django-proxylist-for-grab` into ``urls.py``.. code-block:: pythonurlpatterns = patterns(...url(r'', include('proxylist.urls')),...)`django-proxylist-for-grab` has a list of variables that you can configurethrought django's settings file. You can see the entire list atAdvanced Configuration.Database creation-----------------You have two choices here:Using south~~~~~~~~~~~We ancourage recommend you using `south` for your database migrations. If youalready use it you can migrate `django-proxylist-for-grab`:.. code-block:: bash$ python manage.py migrate proxylistUsing syncdb~~~~~~~~~~~~If you don't want to use `south` you can make a plain *syncdb*:.. code-block:: bash$ python manage.py syncdbBasic setup-----------At first, add a mirror. For working mirror, you need to install app onserver with external ip. This is in order to be able to verify the correctnessof data through proxy server. After adding mirror, you can add and testyour proxies.Asynchronously checking-----------------------`django-proxylist-for-grab` has configured by default to non-async check.You can change this behavior. Insert into your django settings``PROXY_LIST_USE_CALLERY`` and change it to True.After you need to install and configure django-celery and rabbit-mq.For example on OS X~~~~~~~~~~~~~~~~~~~**Packages installation**.. code-block:: bash$ sudo pip install django-celery$ sudo port install rabbitmq-serverAdd the 'djcelery' application to 'INSTALLED_APPS' in settings.. code-block:: pythonINSTALLED_APPS = (...'djcelery',...)**Sync database**.. code-block:: bash$ ./manage.py syncdb**Run rabbitmq and celery**.. code-block:: bash$ sudo rabbitmq-server -detached$ nohup python manage.py celery worker >& /dev/null &Command line reference----------------------update_proxies~~~~~~~~~~~~~~Add new proxies from a file... code-block:: bash$ python manage.py update_proxies [file1] <file2> <...>check_proxies~~~~~~~~~~~~~Check proxies availability and anonymity... code-block:: bash$ python manage.py check_proxiesgrab_proxies~~~~~~~~~~~~Search proxy list on internet.. code-block:: bash$ python manage.py grab_proxiesclean_proxies~~~~~~~~~~~~~Remove broken proxies.. code-block:: bash$ python manage.py clean_proxiesGrabLib usage example:----------------------.. code-block:: pythonfrom proxylist import grabbergrab = grabber.Grab()# Get your ip (You can do this a few times to see how the proxy will be changed)grab.go('http://ifconfig.me/ip')if grab.response.code == 200:print grab.response.body.strip()# Get count of div on google pagegrab.go('http://www.ya.ru/')if grab.response.code == 200:print grab.doc.select('//script').number()GrabLib Spider example:----------------------.. code-block:: python# filename: apps/app/management/commands/spider.py# usage: python manage.py spiderfrom django.core.management.base import BaseCommandfrom grab.spider.base import Taskfrom proxylist.grabber import Spiderclass SimpleSpider(Spider):initial_urls = ['http://www.lib.ru/']def task_initial(self, grab, task):grab.set_input('Search', 'linux')grab.submit(make_request=False)yield Task('search', grab=grab)def task_search(self, grab, task):if grab.doc.select('//b/a/font/b').exists():for elem in grab.doc.select('//b/a/font/b/text()'):print elem.text()class Command(BaseCommand):help = 'Simple Spider'def handle(self, *args, **options):bot = SimpleSpider()bot.run()print bot.render_stats()* GitHub: https://github.com/gotlium/django-proxylist.. image:: https://d2weczhvl823v0.cloudfront.net/gotlium/django-proxylist/trend.png:alt: Bitdeli badge:target: https://bitdeli.com/free