Package: zineb
IntroductionZineb is a lightweight tool solution for simple and efficient web scrapping and crawling built around BeautifulSoup and Pandas. It's main purpose is to helpquickly structure your data in order to be used as fast as possible in data science or machine learning projects.Understanding how Zineb worksZineb gets your custom spider, creates a set ofHTTPRequestobjects for each url, sends the requests and caches a BeautifulSoup object of the page within anHTMLResponseclass of that request.Most of your interactions with the HTML page will be done through theHTMLResponseclass.When the spider starts crawling the page, each response and request in past through the start function:def start(self, response, **kwargs):
     request = kwargs.get('request')
     images = response.imagesGetting startedCreating a projectTo create a project dopython -m zineb startproject <project name>which will create a directory which will have the following structure..myproject
|
|--media
|
|-- models
|-- base.py
|
|--init.py
|
|-- manage.py
|
|-- settings.py
|
|-- spiders.pyOnce the project folder is created, all your interractions with Zineb will be made trough the management commands that are executed throughpython manage.pyfrom your project's directory.The models directory allows you to place the elements that will help structure the data that you have scrapped from from the internet.Themanage.pyfile will allow you to run all the required commands from your project.Finally, the spiders module will contain all the spiders for your project.Configuring your projectOn startup, Zineb implements a set of basic settings (zineb.settings.base) that will get overrided by the values that you would have defined in yoursettings.pylocated in your project.You can read more about this in thesettings section of this file.Creating a spiderCreating a spider is extremely easy and requires a set of starting urls that can be used to scrap one or many HTML pages.class Celebrities(Zineb):
    start_urls = ['http://example.com']

    def start(self, response, request=None, soup=None, **kwargs):
        # Do something hereOnce the Celibrities class is called, each request is passed through thestartmethod. In other words thezineb.http.responses.HTMLResponse,zineb.http.request.HTTPRequestand theBeautifulSoupHTML page object are sent through the function.You are not required to use all these parameters at once. They're just for convinience.In which case, you can also write the start method as so if you only need one of these.def start(self, response, **kwargs):
  # Do something hereOther objects can be passes through the function such as the models that you have created but also the settings of the application etc.Adding meta optionsMeta options allows you to customize certain very specific behaviours [not found in thesettings.pyfile] related to the spider.class Celerities(Zineb):
    start_urls = ['http://example.com']
  
     class Meta:
         domains = []DomainsThis option limits a spider to a very specific set of domains.Verbose nameThis option writter asverbose_namewill specific a different name to your spider.Running commandsStartTriggers the execution of all the spiders present in the given the project. This command will be the main one that you will be using to execute your project.ShellStart a iPython shell on which you can test various elements on the HTML page.When the shell is started, thezineb.http.HTTPRequest, thezineb.response.HTMLResponse, and the BeautifulSoup instance of the page are all injected in the shell.Extractors are passed using aliases:links: LinkExtractorimages: ImageExtractormultilinks: MultiLinkExtractortables: TableExtractorThe extractors are also all passed within the shell in addition to the project settings.In that regards, the shell becomes a interesting place where you can test various querying on an HTML page before using it in your project. For example, using the shell withhttp://example.com.We can get a simple url :IPython 7.19.0

In [1]: response.find("a")
Out[1]: <a href="https://www.iana.org/domains/example">More information...</a>We can find all urls on the page:IPython 7.19.0

In [2]: extractor = links()
In [3]: extractor.resolve(response)
In [4]: str(extrator)
Out [4]: [Link(url=https://www.iana.org/domains/example, valid=True)]

In [5]: response.links
Out [5]: [Link(url=https://www.iana.org/domains/example, valid=True)]Or simply get the page title:IPython 7.19.0

In [6]: response.page_title
Out [6]: 'Example Domain'Remember that in addition to the custom functions created for the class, all the rest called onzineb.response.HTMLResponseare BeautifulSoup functions (find, find_all, find_next...)Queries on the pageLike said previously, the majority of your interactions with the HTML page will be done through theHTMLResponseobject orzineb.http.responses.HTMLResponseclass.This class will implement some very basic general functionnalities that you can use through the course of your project. To illustrate this, let's create a basic Zineb HTTP response from a request:from zineb.http.requests import HTTPRequest

request = HTTPRequest("http://example.com")Requests, when created a not sent [or resolved] automatically if the_sendfunction is not called. In that case, they are marked as being unresolved ex.HTTPRequest("http://example.co", resolved=False).Once the_sendmethod is called, by using thehtml_pageattribute or calling any BeautifulSoup function on the class, you can do all the classic querying on the page e.g. find, find_all...request._send()

request.html_response

    -> Zineb HTMLResponse object

request.html_response.html_page

    -> BeautifulSoup object

request.find("a")

    -> BeautifulSoup TagIf you do not know about BeautifulSoup please readthe documentation here.For instance, suppose you have a spider and want to get the first link present on thehttp://example.compage. That's what you would so:from zineb.app import Zineb

class MySpider(Zineb):
    start_urls = ["http://example.com"]

    def start(self, response=None, request=None, soup=None, **kwargs):
        link = response.find("a")

        # Or, you can also use this tehnic through
        # the request object
        link = request.html_response.find("a")

        # Or you can directly use the soup
        # object as so
        link = soup.find("a")In order to understand what theLink,ImageandTableobjects represents, please read thefollowing sectionof this page.Zineb HTTPRequest objects are better explained in the following section.Getting all the linksrequest.html_response.links

    -> [Link(url=http://example.com valid=True)]Getting all the imagesrequest.html_response.images

    -> [Image(url=https://example.com/1.jpg")]Getting all the tablesrequest.html_response.tables

    -> [Table(url=https://example.com/1")]Getting all the textFinally you can retrieve all the text of the web page at once.request.html_response.text

    -> '\n\n\nExample Domain\n\n\n\n\n\n\n\nExample Domain\nThis domain is for use in   illustrative examples in documents. You may use this\n    domain in literature without prior coordination or asking for permission.\nMore information...\n\n\n\n'ModelsModels are a simple way to structure your scrapped data before saving them to a file. The Model class is built around Panda's excellent DataFrame class in order to simplify as a much as possible the fact of dealing with your data.Creating a custom ModelIn order to create a model, subclass the Model object fromzineb.models.Modeland then add fields to it. For example:from zineb.models.datastructure import Model
from zineb.models.fields import CharField

class Player(Model):
    name = CharField()Using the custom modelOn its own, a model does nothing. In order to make it work, you have to add values to it and then resolve the fields.You can add values to your model in two main ways.Adding a free custom valueThe first method consists of adding values through theadd_valuemethod. This method does not rely on the BeautifulSoup HTML page object which means that values can be added freely.player.add_value('name', 'Kendall Jenner')Adding an expression based valueAddind expression based values requires a BeautifulSoup HTML page object. You can add one value at a time or multiple values.player.add_expression("name", "a#kendall__text", many=True)By using themanyparameter, you can add the all the tags with a specific name and/or attributes to your model at once.Here is a list of expressions that you can use for this field:expressioninterpretationtaga.kendallLink with class kendalla#kendallLind with ID KendallBy default, if a pseudo is not provided,__textpseudo is appended in order to always retrieve the inner text element of the tag.Meta optionsBy adding a Meta to your model, you can pass custom behaviours.OrderingIndexingIndexesOrderingFieldsFields are a very simple way to passing HTML data to your model in a very structured way. Zineb comes with number of preset fields that you can use out of the box:CharFieldTextFieldNameFieldEmailFieldUrlFieldImageFieldIntegerFieldDecimalFieldDateFieldAgeFieldFunctionFieldArrayFieldCommaSeparatedFieldHow fields workOnce the field is called via theresolvefunction on each field which in turns calls thesuper().resolvefunction of theFieldsuper class, the value is stored.By default, the resolve function will do the following things.First, it will run all cleaning functions on the value for example by stripping tags like "<" or ">" by using thew3lib.html.remove_tagslibrary.Second, adeep_cleanmethod will be called on the value which takes out any spaces usingw3lib.html.strip_html5_whitespace, remove escape characters with thew3lib.html.replace_escape_charsfunction and finally reconstruct the value to ensure that any none-detected white space be eliminated.Finally, all validators (default and custom created) are called on the value. The final value is then returned within the model class.CharFieldThe CharField represents the normal character element on an HTML page. You constrain the length.TextFieldThe text field is longer allows you to add paragraphs of text.NameFieldThe name field allows to implement names in your model. Thetitlemethod is called on the string in order to represent the value correctly e.g. Kendall Jenner.EmailFieldThe email field represents emails. The default validator,validators.validate_email, is automatically called on the resolve function fo the class in order to ensure that that the value is indeed an email.UrlFieldThe url field is specific for urls. Just like the email field, the default validator,validators.validate_urlis called in order to validate the url.ImageFieldThe image field holds the url of an image exactly like the UrlField with the sole difference that you can download the image directly when the field is evaluated.class MyModel(Model):
    avatar = ImageField(download=True, download_to="/this/path")IntegerFieldDecimalFieldDateFieldThe date field allows you to pass dates to your model. In order to use this field, you have to pass a date format so that the field can know how to resolve the value.class MyModel(Model):
    date = DateField("%d-%m-%Y")AgeFieldThe age field works likes the DateField but instead of returning the date, it will return the difference between the date and the current date which is an age.FunctionFieldThe function field is a special field that you can use when you have a set of functions to run on the value before returning the final result. For example, let's say you have this valueKendall J. Jennerand you want to run a specific function that takes out the middle letter on every incoming values:def strip_middle_letter(value):
    return

class MyModel(Model):
    name = FunctionField(strip_middle_letter, output_field=CharField(), )Every time the resolve function will be called on this field, the methods provided will be passed on the value.An output field is not compulsory but if not provided, each value will be returned as a character.ArrayFieldAn array field will store an array of values that are all evalutated to an output field that you would have specified.CommaSeperatedFieldCreating your own fieldYou an also create a custom field by suclassingzineb.models.fields.Field. When doing so, your custom field has to provide aresolvefunction in order to determine how the value should be treated. For example:class MyCustomField(Field):
    def resolve(self, value):
        initial_result = super().resolve(value)If you want to use the custom cleaning functionalities on your resolve function before running yours, make sure to call super.ValidatorsValidators make sure that the value that was passed respects the constraints that were implemented as a keyword arguments on the field class. There are five basic validations:Maximum lengthUniquenessNullityDefaultnessValidity (validators)Maximum or Minimum lengthThe maximum length check ensures that the value does not exceed a certain length usingzineb.models.validators.max_length_validatororzineb.models.validators.min_length_validatorwhich are encapsulated and used within thezineb.models.validators.MinLengthValidatororzineb.models.validators.MaxLengthValidatorclass.NullityThe nullity validation ensures that the value is not null and that if a default is provided, that null value be replaced by the latter. It useszineb.models.validators.validate_is_not_null.The defaultness provides a default value for null or none existing ones.Practical examplesFor instance, suppose you want only values that do not exceed a certain length:name = CharField(max_length=50)Or suppose you want a default value for fields that are empty or blank:name = CharField(default='Kylie Jenner')Remember that validators will validate the value itself for example by making sure that an URL is indeed an url or that an email follows the expected pattern that you would expect from an email.Suppose you want only values that would beKendall Jenner. Then you could create a custom validator that would do the following:def check_name(value):
    if value == "Kylie Jenner":
        return None
    return value

name = CharField(validators=[check_name])You can also create validators that match a specific regex pattern using thezineb.models.validators.regex_compilerdecorator:from zineb.models.datastructure import Model
from zineb.models.fields import CharField
from zineb.models.validators import regex_compiler

@regex_compiler(r'\d+')
def custom_validator(value):
    if value > 10:
        return value
    return 0

class Player(Model):
    age = IntegerField(validators=[custom_validator])It is important to understand that the result of the regex compiler is reinjected into your custom validator on which you can then do various other checks.Field resolutionIn order to get the complete structured data, you need to callresolve_valueswhich will return apandas.DataFrameobject:player.add_value("name", "Kendall Jenner")
player.resolve_values()

    -> pandas.DataFramePractically though, you'll be using thesavemethod which also calls theresolve_valuesunder the hood:player.save(commit=True, filename=None, **kwargs)

    -> pandas.DataFrame or new fileBy calling the save method, you'll be able to store the data directly to a JSON or CSV file.ExtractorsExtractors are utilities that facilitates extracting certain specific pieces of data from a web page such as links, images [...] quickly. They can be found inzineb.extactors.Some extractors can be used in various manners. First, with a context processor:extractor = LinkExtractor()
with extractor:
    # Do something hereSecond, in an interation process:for link in extractor:
    # Do something hereFinally, withnext:next(extractor)You can also check if an extractor has a specific value and even concatenate some of them together:# Contains
if x in extractor:
    # Do something here

# Addition
concatenated_extractors = extractor1 + extractor2LinkExtractorurl_must_contain- only keep urls that contain a specific stringunique- return a unique set of urls (no duplicates)base_url- reconcile a domain to a pathonly_valid_links- only keep links (Link) that are marked as validextractor = LinkExtractor()
extractor.finalize(response.html_response)

    -> [Link(url=http://example.com, valid=True)]There might be times where the extracted links are relative paths. This can cause an issue for running additional requests. In which case, use thebase_urlparameter:extractor = LinkExtractor(base_url=http://example.com)
extractor.finalize(response.html_response)

# Instead of getting this result which would also
# be marked as a none valid link

    -> [Link(url=/relative/path, valid=False)]

# You will get the following with the full url link

    -> [Link(url=http://example.com/relative/path, valid=True)]NOTE: By definition, a relative path is not a valid link hence the valid set to False.MultiLinkExtractorAMultiLinkExtractorworks exactly like theLinkExtractorwith the only difference being that it also identifies and collects emails that are contained within the HTML page.TableExtractorExtract all the rows from the first table that is matched on the HTML page.class_name- intercept a table with a specific class namehas_headers- specify if the table has headers in order to ignore it in the final datafilter_empty_rows- ignore any rows that do not have a valuesprocessors- a set of functions to run on the data once it is all extractedImageExtractorExtract all the images on the HTML page.You can filter down the images that you get by using a specific set of parameters:unique- return only a unique et set of urlsas_type- only return images having a specific extensionurl_must_contain- only return images which contains a specific stringmatch_height- only return images that match as specific heightmatch_width- only return images that match a specific widthTextExtractorExtract all the text on an HTML page.First, the text is retrieved as a raw value then tokenized and vectorized usingnltk.tokenize.PunktSentenceTokenizerandnltk.tokenize.WordPunctTokenizer.To know more about NLKT,please read the following documentation.Zineb special wrappersHTTPRequestZineb uses a special built-in HTTPRequest class which wraps the following for better cohesion:Therequests.Requestresponse classThebs4.BeautifulSoupobjectIn general, you will not need to interact with this class that much because it's just an interface for implement additional functionnalities especially to the Request class.follow: create a new instance of the class whose resposne will be the one of a new urlfollow_all: create new instances of the class who responses will tbe the ones of the new urlsurljoin: join a path the domainHTMLResponseIt wraps the BeautifulSoup object in order to implement some small additional functionalities:page_title: return the page's titlelinks: return all the links of the pageimages: return all the images of the pagetables: return all the tables of the pageSignalsSignals are a very simple yet efficient way for you to run functions during the lifecycle of your project when certain events occur at very specific moments.Internally signals are sent on the following events:When the registry is populatedBefore the spider startsAfter the spider has startedBefore an HTTP request is sentBefore and HTTP request is sentBefore the model downloads anythingAfter the model has downloaded somethingCreating a custom signalTo create custom signal, you need to mark a method as being a receiver for any incoming signals. For example, if you want to create a signal to intercept one of the events above, you should do:from zineb.signals import receiver

@receiver(tag="Signal Name")
def my_custom_signal(sender, **kwargs):
    passThe signals function has to be able to accept asenderobject and additional parameters such as the current url or the current HTML page.You custom signals do not have to return anything.PipelinesPipelines are a great way to send chained requests to the internet or treat a set of responses by processing them afterwards through a set of functions of your choice.Some Pipeplines are also perfect for donwloading images.ResponsesPipelineThe response pipepline allows you to chain a group of responses and treat all of them at once through a function:from zineb.http.pipelines import ResponsesPipeline

pipeline = ResponsesPipeline([response1, response2], [function1, function2])
pipeline.results
    -> listIt comes with three main parameters:responses- which corresponds to a list of HTMLResponsesfunctions- a list of functions to pass each individual response and additional parametersparamaters- a set of additional parameters to pass to the functionsThe best way to use the ResponsesPipeline is within the functions of your custom spider:class MySpider(Zineb):
   start_urls = ["https://example.com"]

   def start(self, response, soup=None, **kwargs):
       extractor = LinksExtractor()
       extractor.resolve(soup)
       responses = request.follow_all(*list(extractor))
       ResponsesPipeline(responses, [self.do_something_here])

   def do_something_here(self, response, soup=None, **kwargs):
       # Continue parsing data hereN.B.Each function is executed sequentially. So, the final result will come from the final function of the listHTTPPipelineThis pipeline takes a set of urls, creates HTTPResquests for each of them and then sends them to the internet.If you provided a set of functions, it will pass each request through them.from zineb.http.pipelines import HTTPPipeline
from zineb.utils.general import download_image

HTTPPipeline([https://example.com], [download_image])Each function should be able to accept an HTTP Response object.You can also pass additional parameters to your functions by doing the following:HTTPPipeline([https://example.com], [download_image], parameters={'extra': False})In this specific case, your function should accept anextraparameter which result would be False.CallbackThe Callback class allows you to run a callback function once each url is processed and passed through the main start function of your spider.The__call__method is triggerd on the instance in order to resolve the function to use.class Spider(Zineb):
    start_urls = ["https://example.com"]

    def start(self, response, **kwargs):
        request = kwargs.get("request")
        model = MyModel()
        return Callback(request.follow, self.another_function, model=model)

    def another_function(self, response, **kwargs):
        model = kwargs.get("model")
        model.add_value("name", "Kendall Jenner")
        model.save()UtilitiesLink reconciliationMost of times, when you retrieve links from a page, they are returned as relative paths. Theurljoinmethod reconciles the url of the visited page with that path.<a href="/kendall-jenner">Kendall Jenner</a>

# Now we want to reconcile the relative path from this link to
# the main url that we are visiting e.g. https://example.com

request.urljoin("/kendall-jenner")

-> https://example.com/kendall-jennerSettingsThis section will talk about all the available settings that are available for your project and how to use them for web scrapping.PROJECT_PATHRepresents the current path for your project. This setting is not be changed.SPIDERSIn order for your spider to be executed, every created spider should be registered here. The name of the class should serve as the name of the spider to be used.SPIDERS = [
    "MySpider"
]DOMAINSYou can restrict your project to use only to a specific set of domains by ensuring that no request is sent if it matches one of the domains within this list.DOMAINS = [
    "example.com"
]ENSURE_HTTPSEnforce that every link in your project is a secured HTTPS link. This setting is set to False by default.MIDDLEWARESMiddlewares are functions/classes that are executed when a signal is sent from any part of the project. Middlewares implement extra functionnalities without affecting the core parts of the project. They can then be disabled safely if you do not need them.MIDDLEWARES = [
    "zineb.middlewares.handlers.Handler",
    "myproject.middlewares.MyMiddleware"
]The main Zineb middlewares are the following:zineb.middlewares.referer.Refererzineb.middlewares.handlers.Handlerzineb.middlewares.automation.Automationzineb.middlewares.history.Historyzineb.middlewares.statistics.GeneralStatisticszineb.middlewares.wireframe.WireFrameUSER_AGENTSA user agent is a characteristic string that lets servers and network peers identify the application, operating system, vendor, and/or version of the requestingMDN Web Docs.Implement additional sets of user agents to your projects in addition to those that were already created.RANDOMIZE_USER_AGENTSSpecifies whether to use one user agent for every request or to randomize user agents on every request. This setting is set to to False by default.DEFAULT_REQUEST_HEADERSSpecify additional default headers to use for each requests.The default initial headers are:Accept-Language- enAccept- text/html,application/json,application/xhtml+xml,application/xml;q=0.9,/;q=0.8Referrer - NonePROXIESUse a set of proxies for each request. When a request in sent, a random proxy is selected and implemented with the request.PROXIES = [
    ("http", "127.0.0.1"),
    ("https", "127.0.0.1")
]RETRYSpecifies the retry policy. This is set to False by default. In other words, the request silently fails and never retries.RETRY_TIMESSpecificies the amount of times the the request is sent before eventually failing.RETRY_HTTP_CODESIndicates which status codes should trigger a retry. By default, the following codes: 500, 502, 503, 504, 522, 524, 408 and 429 will trigger it.