Package: pymeilisearch-scraper
pymeilisearch-scraperAn Ansys fork of meilisearch/docs-scraperThis repository has been forked frommeilisearch/docs-scraperand
incorporates several enhancements to facilitate usage with Python and Sphinx
documentation scraping.It is used bypymeilisearchwhen
scrapping online and local documentation pages.Added:Ability to install viapipIgnore" #"at the end of headers for sphinx documentationAdded a__main__.pyto allow you to call this as a Python moduleIncludes desired CNAME when scrapping local pages$ python -m scraper -h
usage: __main__.py [-h] [--meilisearch-host-url MEILISEARCH_HOST_URL]
                   [--meilisearch-api-key MEILISEARCH_API_KEY]
                   config_file

Scrape documentation.

positional arguments:
  config_file           The path to the configuration file.

options:
  -h, --help            show this help message and exit
  --meilisearch-host-url MEILISEARCH_HOST_URL
                        The URL to the meilisearch host
  --meilisearch-api-key MEILISEARCH_API_KEY
                        The URL to the meilisearch hostOriginal documentation follows:docs-scraperMeilisearch|Meilisearch Cloud|Documentation|Discord|Roadmap|Website|FAQdocs-scraperis a scraper for your documentation website that indexes the scraped content into aMeilisearchinstance.Meilisearchis an open-source search engine.Discover what Meilisearch is!This scraper is used in production and runs on theMeilisearch documentationon each deployment.💡 If you already have your own scraper but you still want to use Meilisearch and ourfront-end tools, check outthis discussion.Table of Contents⚡ Supercharge your Meilisearch experience⚙️ UsageRun your Meilisearch InstanceSet your Config FileRun the Scraper🖌 And for the front-end search bar?🛠 More ConfigurationsMore About the SelectorsAll the Config File Settingsindex_uidstart_urlsstop_urls(optional)selectors_key(optional)scrape_start_urls(optional)sitemap_urls(optional)sitemap_alternate_links(optional)selectors_exclude(optional)custom_settings(optional)min_indexed_level(optional)only_content_level(optional)js_render(optional)js_wait(optional)allowed_domains(optional)AuthenticationInstalling Chrome Headless🤖 Compatibility with Meilisearch⚙️ Development Workflow and ContributingCredits⚡ Supercharge your Meilisearch experienceSay goodbye to server deployment and manual updates withMeilisearch Cloud. No credit card required.⚙️ UsageHere are the 3 steps to usedocs-scraper:Run a Meilisearch instanceSet your config fileRun the scraperRun your Meilisearch InstanceYour documentation content needs to be scraped and pushed into a Meilisearch instance.You can install and run Meilisearch on your machine usingcurl.curl-Lhttps://install.meilisearch.com|sh
./meilisearch--master-key=myMasterKeyThere areother ways to install Meilisearch.The host URL and the API key you will provide in the next steps correspond to the credentials of this Meilisearch instance.
In the example above, the host URL ishttp://localhost:7700and the API key ismyMasterKey.Meilisearch is open-source and can run either on your server or on any cloud provider. Here is a tutorial torun Meilisearch in production.Set your Config FileThe scraper tool needs a config file to know which content you want to scrape. This is done by providingselectors(e.g. the HTML tag/id/class). The config file is passed as an argument. It follows no naming convention and may be named as you want.Here is an example of a basic config file:{"index_uid":"docs","start_urls":["https://www.example.com/doc/"],"sitemap_urls":["https://www.example.com/sitemap.xml"],"stop_urls":[],"selectors":{"lvl0":{"selector":".docs-lvl0","global":true,"default_value":"Documentation"},"lvl1":{"selector":".docs-lvl1","global":true,"default_value":"Chapter"},"lvl2":".docs-content .docs-lvl2","lvl3":".docs-content .docs-lvl3","lvl4":".docs-content .docs-lvl4","lvl5":".docs-content .docs-lvl5","lvl6":".docs-content .docs-lvl6","text":".docs-content p, .docs-content li"}}Theindex_uidfield is the index identifier in your Meilisearch instance in which your website content is stored. The scraping tool will create a new index if it does not exist.Thedocs-contentclass (the.means this is a class) is the main container of the textual content in this example. Most of the time, this tag is a<main>or an<article>HTML element.lvlXselectors should use the standard title tags likeh1,h2,h3, etc. You can also use static classes. Set a unique id or name attribute to these elements.Every searchablelvlelements outside this main documentation container (for instance, in a sidebar) must beglobalselectors. They will be globally picked up and injected to every document built from your page.You can also check out theconfig filewe use in production for our own documentation site.💡To better understand the selectors, go tothis section.🔨There are many other fields you can set in the config file that allow you to adapt the scraper to your need. Check outthis section.Run the ScraperFrom Source CodeThis project supports Python 3.8 and above.Thepipenvcommandmust be installed.Set both environment variablesMEILISEARCH_HOST_URLandMEILISEARCH_API_KEY.Following on from the example in thefirst step, they are respectivelyhttp://localhost:7700andmyMasterKey.Then, run:pipenvinstall
pipenvrun./docs_scraper<path-to-your-config-file><path-to-your-config-file>should be the path of your configuration file defined at theprevious step.With Dockerdockerrun-t--rm\-eMEILISEARCH_HOST_URL=<your-meilisearch-host-url>\-eMEILISEARCH_API_KEY=<your-meilisearch-api-key>\-v<absolute-path-to-your-config-file>:/docs-scraper/<path-to-your-config-file>\getmeili/docs-scraper:latestpipenvrun./docs_scraper<path-to-your-config-file><absolute-path-to-your-config-file>should be the absolute path of your configuration file defined at theprevious step.⚠️ If you run Meilisearch locally, you must add the--network=hostoption to this Docker command.In a GitHub ActionTo run after your deployment job:run-scraper:
    needs: <your-deployment-job>
    runs-on: ubuntu-18.04
    steps:
    - uses: actions/checkout@master
    - name: Run scraper
      env:
        HOST_URL: ${{ secrets.MEILISEARCH_HOST_URL }}
        API_KEY: ${{ secrets.MEILISEARCH_API_KEY }}
        CONFIG_FILE_PATH: <path-to-your-config-file>
      run: |
        docker run -t --rm \
          -e MEILISEARCH_HOST_URL=$HOST_URL \
          -e MEILISEARCH_API_KEY=$API_KEY \
          -v $CONFIG_FILE_PATH:/docs-scraper/<path-to-your-config-file> \
          getmeili/docs-scraper:latest pipenv run ./docs_scraper <path-to-your-config-file>⚠️ We do not recommend using thelatestimage in production. Use therelease tagsinstead.Here is theGitHub Action filewe use in production for the Meilisearch documentation.About the API KeyThe API key you must provide should have the permissions to add documents into your Meilisearch instance.In a production environment, we recommend providing the private key instead of the master key, as it is safer and it has enough permissions to perform such requests._More aboutMeilisearch authentication. _🖌 And for the front-end search bar?After having scraped your documentation, you might need a search bar to improve your user experience!About the front part:If your website is a VuePress application, check out thevuepress-plugin-meilisearchrepository.For all kinds of documentation, check out thedocs-searchbar.jslibrary.Both of these libraries provide a front-end search bar perfectly adapted for documentation.🛠 More ConfigurationsMore About the SelectorsBasesVery simply, selectors are needed to tell the scraper "I want to get the content in this HTML tag".This HTML tag is aselector.A selector can be:a class (e.g..main-content)an id (e.g.#main-article)an HTML tag (e.g.h1)With a more concrete example:"lvl0":{"selector":".navbar-nav .active","global":true,"default_value":"Documentation"},.navbar-nav .activemeans "take the content in the classactivethat is itself in the classnavbar-nav".global: truemeans you want the samelvl0(so, the same main title) for all the contents extracted from the same page."default_value": "Documentation"will be the displayed value if no content in.navbar-nav .activewas found.NB: You can set theglobalanddefault_valueattributes for every selector level (lvlX) and not only for thelvl0.The LevelsYou can notice different levels of selectors (0 to 6 maximum) in the config file. They correspond to different levels of titles, and will be displayed this way:Your data will be displayed with a main title (lvl0), sub-titles (lvl1), sub-sub-titles (lvl2) and so on...All the Config File Settingsindex_uidTheindex_uidfield is the index identifier in your Meilisearch instance in which your website content is stored. The scraping tool will create a new index if it does not exist.{"index_uid":"example"}start_urlsThis array contains the list of URLs that will be used to start scraping your website.The scraper will recursively follow any links (<a>tags) from those pages. It will not follow links that are on another domain.{"start_urls":["https://www.example.com/docs"]}Using Page RankThis parameter gives more weight to some pages and helps to boost records built from the page.Pages with highestpage_rankwill be returned before pages with a lowerpage_rank.{"start_urls":[{"url":"http://www.example.com/docs/concepts/","page_rank":5},{"url":"http://www.example.com/docs/contributors/","page_rank":1}]}In this example, records built from the Concepts page will be ranked higher than results extracted from the Contributors page.stop_urls(optional)The scraper will not follow links that matchstop_urls.{"start_urls":["https://www.example.com/docs"],"stop_urls":["https://www.example.com/about-us"]}selectors_key(optional)This allows you to use custom selectors per page.If the markup of your website is so different from one page to another that you can't have generic selectors, you can namespace your selectors and specify which set of selectors should be applied to specific pages.{"start_urls":["http://www.example.com/docs/",{"url":"http://www.example.com/docs/concepts/","selectors_key":"concepts"},{"url":"http://www.example.com/docs/contributors/","selectors_key":"contributors"}],"selectors":{"default":{"lvl0":".main h1","lvl1":".main h2","lvl2":".main h3","lvl3":".main h4","lvl4":".main h5","text":".main p"},"concepts":{"lvl0":".header h2","lvl1":".main h1.title","lvl2":".main h2.title","lvl3":".main h3.title","lvl4":".main h5.title","text":".main p"},"contributors":{"lvl0":".main h1","lvl1":".contributors .name","lvl2":".contributors .title","text":".contributors .description"}}}Here, all documentation pages will use the selectors defined inselectors.defaultwhile the page under./conceptswill useselectors.conceptsand those under./contributorswill useselectors.contributors.scrape_start_urls(optional)By default, the scraper will extract content from the pages defined instart_urls. If you do not have any valuable content on your starts_urls or if it's a duplicate of another page, you should set this tofalse.{"scrape_start_urls":false}sitemap_urls(optional)You can pass an array of URLs pointing to your sitemap(s) files. If this value is set, the scraper will try to read URLs from your sitemap(s){"sitemap_urls":["http://www.example.com/docs/sitemap.xml"]}sitemap_alternate_links(optional)Sitemaps can contain alternative links for URLs. Those are other versions of the same page, in a different language, or with a different URL. By default docs-scraper will ignore those URLs.Set this to true if you want those other versions to be scraped as well.{"sitemap_urls":["http://www.example.com/docs/sitemap.xml"],"sitemap_alternate_links":true}With the above configuration and thesitemap.xmlbelow, bothhttp://www.example.com/docs/andhttp://www.example.com/docs/de/will be scraped.<url><loc>http://www.example.com/docs/</loc><xhtml:linkrel="alternate"hreflang="de"href="http://www.example.com/de/"/></url>selectors_exclude(optional)This expects an array of CSS selectors. Any element matching one of those selectors will be removed from the page before any data is extracted from it.This can be used to remove a table of content, a sidebar, or a footer, to make other selectors easier to write.{"selectors_exclude":[".footer","ul.deprecated"]}custom_settings(optional)This field can be used to add Meilisearch settings.Example:"custom_settings":{"synonyms":{"static site generator":["ssg"],"ssg":["static site generator"]},"stopWords":["of","the"],"filterableAttributes":["genres","type"]}Learn more aboutfilterableAttributes,synonyms,stop-wordsand all available settings in theMeilisearch documentation.min_indexed_level(optional)The default value is 0. By increasing it, you can choose not to index some records if they don't have enoughlvlXmatching. For example, with amin_indexed_level: 2, the scraper indexes temporary records having at least lvl0, lvl1 and lvl2 set.This is useful when your documentation has pages that share the samelvl0andlvl1for example. In that case, you don't want to index all the shared records, but want to keep the content different across pages.{"min_indexed_level":2}only_content_level(optional)Whenonly_content_levelis set totrue, then the scraper won't create records for thelvlXselectors.If used,min_indexed_levelis ignored.{"only_content_level":true}js_render(optional)Whenjs_renderis set totrue, the scraper will use ChromeDriver. This is needed for pages that are rendered with JavaScript, for example, pages generated with React, Vue, or applications that are running in development mode:autoreloadwatch.After installing ChromeDriver, provide the path to the bin using the following environment variableCHROMEDRIVER_PATH(default value is/usr/bin/chromedriver).The default value ofjs_renderisfalse.{"js_render":true}js_wait(optional)This setting can be used whenjs_renderis set totrueand the pages need time to fully load.js_waittakes an integer is specifies the number of seconds the scraper should wait for the page to load.{"js_render":true,"js_wait":1}allowed_domains(optional)This setting specifies the domains that the scraper is allowed to access. In most cases theallowed_domainswill be automatically set using thestart_urlsandstop_urls. When scraping a domain that contains a port, for examplehttp://localhost:8080, the domain needs to be manually added to the configuration.{"allowed_domains":["localhost"]}AuthenticationWARNING:Please be aware that the scraper will send authentication headers to every scraped site, so useallowed_domainsto adjust the scope accordingly!Basic HTTPBasic HTTP authentication is supported by setting these environment variables:DOCS_SCRAPER_BASICAUTH_USERNAMEDOCS_SCRAPER_BASICAUTH_PASSWORDCloudflare Access: Identity and Access ManagementIf it happens to you to scrape sites protected by Cloudflare Access, you have to set appropriate HTTP headers.Values for these headers are taken from env variablesCF_ACCESS_CLIENT_IDandCF_ACCESS_CLIENT_SECRET.In case of Google Cloud Identity-Aware Proxy, please specify these env variables:IAP_AUTH_CLIENT_ID- # pickclient ID of the applicationyou are connecting toIAP_AUTH_SERVICE_ACCOUNT_JSON- # generate inActions-> Create key -> JSONKeycloak Access: Identity and Access ManagementIf you need to scrape site protected byKeycloak(Gatekeeper), you have to provide a valid access token.If you set the environment variablesKC_URL,KC_REALM,KC_CLIENT_ID, andKC_CLIENT_SECRETthe scraper authenticates itself against Keycloak usingClient Credentials Grantand adds the resulting access token asAuthorizationHTTP header to each scraping request.Installing Chrome HeadlessWebsites that need JavaScript for rendering are passed through ChromeDriver.Download the versionsuited to your OS and then set the environment variableCHROMEDRIVER_PATH.🤖 Compatibility with MeilisearchThis package guarantees compatibility withversion v1.x of Meilisearch, but some features may not be present. Please check theissuesfor more info.⚙️ Development Workflow and ContributingAny new contribution is more than welcome in this project!If you want to know more about the development workflow or want to contribute, please visit ourcontributing guidelinesfor detailed instructions!CreditsBased onAlgolia's docsearch scraper repositoryfromthis commit.Due to a lot of future changes on this repository compared to the original one, we don't maintain it as an official fork.Meilisearchprovides and maintains manySDKs and Integration toolslike this one. We want to provide everyone with anamazing search experience for any kind of project. If you want to contribute, make suggestions, or just know what's going on right now, visit us in theintegration-guidesrepository.