Package: olxsearch
Download data from OLX listingsThis is a Python script that can download listings from the small ads platform OLX in the following countries:ArgentinaBulgariaBosniaBrazilColombiaCosta RicaEcuadorEgyptGuatemalaIndiaIndonesiaKazakhstanLebanonOmanPakistanPanamaPeruPolandPortugalRomaniaSan SalvadorSouth AfricaUkraineUzbekistanIf you useolxsearchfor scientific research, please cite it in your publication:Fink, C. (2020):olxsearch: Python script to download OLX small ads data.doi:10.5281/zenodo.3906038.DependenciesThe script is written in Python 3 and depends on the Python modulesBeautifulSoup,dateparser,geocoder,pandasandRequests.To install dependencies on a Debian-based system, run:apt-getupdate-y&&apt-getinstall-ypython3-devpython3-pippython3-virtualenv(There’s an Archlinux AUR package pulling in all dependencies, see further down)Installationusingpipor similar:pip3installolxsearchOR: manually:Clone this repositorygitclonehttps://gitlab.com/christoph.fink/olxsearch.gitChange to the cloned directoryUse the Pythonsetuptoolsto install the package:cdolxsearch
python3./setup.pyinstallOR: (Arch Linux only) fromAUR:# e.g. using yayyaypython-olxsearchUsageImport theolxsearchmodule.Then instantiate anolxsearch.OlxSearchobject, supplying acountryand asearch_termas arguments. The object’slistingsproperty is a generator providing access to each ad listed on the platform that matches the supplied search term. Itslistings_as_dataframeproperty is apandas.DataFramecontaining all these ads.importolxsearcholx_search_argentina=olxsearch.OlxSearch("Argentina","Yerba mate")print(next(olx_search_argentina.listings))# {'id': '1102114778', 'title': 'YERBA MATE SECADERO X 500 GRS.', 'description': 'YERBA MATE SECADERO \nPAQUETE X 500 GRS. $70\nPACK X 10 UNIDADES VENTA MÍNIMA\nCALIDAD DE EXPORTACIÓN \nEXCELENTE RELACIÓN PRECIO * CALIDAD \nAPROVECHE ANTES QUE SE TERMINEN\nCOMUNÍQUESE A NUESTRO WHATSAPP', 'created_at': '2020-02-18T16:57:38-03:00', 'created_at_first': '2020-02-18T16:57:02-03:00', 'republish_date': None, 'images': ['https://apollo-virginia.akamaized.net:443/v1/files/ns52s6zc369y2-AR/image'], 'price': (70.0, 'ARS'), 'lat': -34.626, 'lon': -58.4}# pandas DataFrameolx_search_southafrica=olxsearch.OlxSearch("South Africa","Biltong")listings=olx_search_southafrica.listings_as_dataframe#             id                                              title  ...        lat        lon# 0   1061464181                                     Biltong slicer  ... -25.703179  28.178248# 1   1061707900         Claasen Biltong Slicer excellent condition  ... -28.549999  25.233299# 2   1061884723                                      Biltong maker  ... -26.701476  27.092649# ...# 38  1061429395                                      Biltong snyer  ... -29.082081  26.148292# 39  1059714562  Biltongkas / biltong box / biltong dryers / me...  ... -25.712152  28.002048## [40 rows x 10 columns]Data privacyBy default, olxsearch pseudonymises downloaded metadata, i.e. it replaces (direct) identifiers with randomised identifiers (generated using hashes, i.e. one-way “encryption”). This serves as one step of a responsible data processing workflow. However, other (meta-)data might nevertheless qualify asindirect identifiers, as they, combined or on their own, might allow re-identification of the seller. If you want to use data downloaded using olxsearch in a GDPR-compliant fashion, you have to follow up the data collection stage withdata minimisationand further pseudonymisation or anonymisation efforts.Olxsearch can keep original identifiers (i.e. skip pseudonymisation). To instruct it to do so, instantiate anOlxSearchwith the parameterpseudonymise_identifiers=False. Ensure that you fulfil all legal and organisational requirements to handle personal information before you decide to collect non-pseudonyismed data.importolxsearchdownloader=OlxSearch("Ecuador","bolones verdes",pseudonymise_identifiers=False# get legal/ethics advice before doing this)