Saravanan666/pypi-org huggingface dataset:
Package: site-configuration-client
Site Configuration ClientREST API client for the Site Configuration service.CompatibilityThis package tested with both Python 3.9 and Python 3.5.It has few required and optional dependencies:requests:Required. Explicitly specified insetup.pydjango:Optional. Tested to work with both Django 3 and
Django 2.google-cloud-storage:Optional. Tested with version 1.x.For the exact versions please refer to bothsetup.pyand therequirementsdirectory.InstallationInstall thesite-configuration-clientdependency from PyPi either viaserver-vars.ymlor other build processes such as Docker.Add the following settings toserver-vars.yml(orlms.ymldepending on your deployment configuration).SITE_CONFIG_BASE_URL:"https://siteconfig:14000/"SITE_CONFIG_API_TOKEN:"apitokengoeshere"SITE_CONFIG_ENVIRONMENT:"development"
Package: site-conn-check
Site-Conn-CheckA CLI program that requests the header of a website to determine its Status Code and response time and visualizes it in a dataframe.
If you find any issues feel free to drop them in the issues section.Set Uppippip install Site-Conn-CheckClone and buildAfter cloning to build the project just run:$ py -m build --wheel$ pip install <.whl file>required packages:BuildCommandsNOTE: Upon adding you first site a database file will be created. Running without adding a site will likely result in errors.This will happen everytime you delete or lose your database file.$ site_conn_check add -n <SiteName> -u <URL>- Adds a new Site to the database.$ site_conn_check remove -n <SiteName>- Removes a Site from the database.$ site_conn_check update -n <SiteName>- Updates a Site.$ site_conn_check -s- Lists all Sites in the database.$ site_conn_check -d- Deletes database file.$ site_conn_check --run- Executes status and response time collection and displays it in a styled table.
Package: sitecrawl
Simple Python module to crawl a website and extract URLs.InstallationUsing pip:pip3installsitecrawlsitecrawl--helpOr build from sources:# Clone projectgitclonehttps://github.com/gabfl/sitecrawl&&cdsitecrawl# Installationpip3install.UsageCLIsitecrawl--urlhttps://www.yahoo.com/--depth2--max4--verbose->* Found 4 internal URLs
  https://www.yahoo.com
  https://www.yahoo.com/entertainment
  https://www.yahoo.com/lifestyle
  https://www.yahoo.com/plus

* Found 5 external URLs
  https://mail.yahoo.com/
  https://news.yahoo.com/
  https://finance.yahoo.com/
  https://sports.yahoo.com/
  https://shopping.yahoo.com/

* Skipped 0 URLsAs a moduleBasic example:fromsitecrawlimportcrawlcrawl.base_url='https://www.yahoo.com'crawl.deep_crawl(depth=2)print('Internal URLs:',crawl.get_internal_urls())print('External URLs:',crawl.get_external_urls())print('Skipped URLs:',crawl.get_skipped_urls())A more detailed example is available inexample.py.
Package: sitedeploy
sitedeployis a replacement for sitecopy.supported protocols: FTP, SSH (viaparamiko)fast synchronization~/.sitedeployrcexample:site bigcompany
  server ftp.bigcompany.com
  remote /bigcompanysite.com/www
  local /var/www/big
  protocol ftp
  username foo
  password xxxx
  permissions all
  exclude templates_c
  exclude *.log

site smallcompany
  server ftp.smallc.com
  remote /html/
  local /var/www/small
  protocol ssh
  permissions all
  username bar
  password xxxxBasic steps:initialize (need to be performed only once):$ sitedeploy.py -i bigcompanyupdate changes:$ sitedeploy.py -u bigcompany
Package: sitefab
SiteFab: state of the art static website generator for humansGetting Started in 30 secondsHere is how to install SiteFab, Create a new site and compile it in less than
30 seconds:# install the main packagepipinstallsitefab
spacydownloaden_core_web_sm
pipinstall-Uspacy-lookups-data# clone the site template as basegitclonehttps://github.com/ebursztein/sitefab-template.gitmysite/# clone the plugins in your site directorygitclonehttps://github.com/ebursztein/sitefab-plugins.gitmysite/plugins# generate your shiny new sitesitefab-cmysite/config/sitefab.yamlgenerateEt voila! you know have a fully fonctional site that can be statically compiled.
As the next step you should add your content, customize the html templates and
tweak the plugins configuration to your liking!Key FeaturesHere are some of the key highlevel functionalities that makes SiteFab the static generator of choice for modern websites:State of the art images processing stack: The images stack generate out of the box all you need to use image in optimal fashion including: reactive images for the<picture>element, webp versions for the browsers who support it, frozen thumbnail for the loading and cropped thumbnails for previews.Bleeding edge natural language processing stack: Leveraging the latest advance in NLP Sitefab offers: An accurate list of related posts (LSI algorithm), a client side faceted search and a smart client-side autocompletionFully customizable: Itsflexible plugin systemandtemplatized parsermake it easy and fast to customize SiteFab to your needs.Design principlesBeside its technical features what separates SiteFab from other site generators
is its guiding principles:Configuration over convention: Every behavior is explictly specified.
There are no special files or directories. SiteFab only do what the
site configuration say. Nothing more, nothing less.Content Agnostic: There is only one type of content called post.
SiteFab makes no assumption about the content stored in the .md files. Its
job is to apply the configuration specified in the frontmatter to render
the specificied files and ensure that the enabled plugins are properly
executed.Atomicity: Each piece of content is self contained by having its own
configuration in its frontmatter including which template to use.
This allows to accomodate both complex sites with per page configuration
and very simple ones that reuse the same template again and again.Orthogonality: Site configuration, plugins and plugins configuration
are independent of the core engine so you can use as many configurations
and tweaked plugins you want. Additionally you can check those alongside
with your site content to have reproducible build, safe rollbacks and
concurrent version of the build pipeline (e.g alpha versus stable).AlternativesHere are some popular alternatives if SiteFab is not what you are looking for:Jekyll: The most popular site generate, written inRuby.Hugo: A popular site generator, written inGothat
focuses on speed.Pelican: Another site generator written
inPython.
Package: sitejabber-utils
Python UtilsLibrary for user's payload encryption used with Sitejabber.InstallationInstallation is easy using pip. Just run the following on the command line:pip install sitejabber-utilsUsageGet your API keys onhttps://biz.sitejabber.com/accountJSON encode your user dataCall the encrypt methodURL-encode the resultRedirect the user to the feedback linkfrom json import dumps
from sitejabber import utils
try:
	from urllib.parse import quote_plus  # Python 3+
except ImportError:
	from urllib import quote_plus  # Python 2.X

userData = {
	"email":	"janedoe@gmail.com",
	"order_date":	"06-13-2013",
	"order_id":	"1234",
	"first_name":	"Jane",
	"last_name":	"Doe"
}

encryptedData = utils.encrypt(dumps(userData), API_SECRET)
feedbackLink = "https://www.sitejabber.com/biz-review?key=API_KEY&payload=" + quote_plus(encryptedData);
print(feedbackLink)
Package: sitemap
Sitemap library for dynamically generatingsitemap.xmlInstallationpipinstallsitemapUsageUrlsetfromsitemapimportUrl,Urlseturlset=Urlset()url=Url('https://www.example.com/',changefreq='weekly')urlset.add_url(url)# urlset.to_string()urlset.write_xml('sitemap.xml')SiteindexfromsitemapimportSitemap,Siteindexsiteindex=Siteindex()sitemap=Sitemap('https://www.example.com/sitemap.xml')siteindex.add_sitemap(sitemap)# siteindex.to_string()siteindex.write_xml('sitemap.xml')
Package: sitemap3
[![Build Status](https://travis-ci.org/konstantinfarrell/sitemap3.svg?branch=develop)](https://travis-ci.org/konstantinfarrell/sitemap3)
[![Coverage Status](https://coveralls.io/repos/github/konstantinfarrell/sitemap3/badge.svg?branch=develop)](https://coveralls.io/github/konstantinfarrell/sitemap3?branch=develop)# Sitemap3Sitemap3 is an async sitemap generation utility written in Python 3.5.
It currently generates a list of urls and either outputs them or writes them to a file.## Installpip install sitemap3orgit clonegit@github.com:konstantinfarrell/sitemap3.git
cd sitemap3
python setup.py installDon’t forget a virtualenv## UsageFrom the shellsitemap –u <url>
sitemap –u <url> –w <output>Within a scriptfrom sitemap import sitemapurl = ‘https://konstantinfarrell.github.io’
results = sitemap(url)## TODOAdd xml support## KNOWN BUGSLarger sites may cause program to timeout
Package: sitemap42
Installation$[sudo]pipinstallsitemap42Examplesfromsitemap42importSitemapsitemap=Sitemap()sitemap.append('https://domain.com/page1',)sitemap.append('https://domain.com/page2',changefreq='daily',priority=0.5,lastmod=datetime.now())sitemap.tostring()sitemap.write('sitemap.xml')fromsitemap42importSitemapindexsitemapindex=Sitemap()sitemapindex.append('https://domain.com/sitemap1.xml')sitemapindex.append('https://domain.com/sitemap2.xml',lastmod=datetime.now())LinksSitemaps XML formatreadme42.com
Package: sitemapa
SitemapaCreate advanced sitemaps easily.Sitemapa is a small package to reduce your work while generating sitemaps. You describe your sitemaps with JSON-structure. Sitemapa is framework agnostic and not indexing your website — it's just generating sitemaps from your description. Noting more. I use it to generate sitemaps for millions of URLs on my websites.Keep in mind that it's your job to validate your urls and lastmod dates.Learn more about sitemaps— it's important for your website SEO.FeaturesNo extra dependenciesCreate regular sitemaps. URLs, Images, Videos and News are supported.Create index sitemap to combine your regular sitemapsCreate extra attributes for your tags like <video:restriction relationship="allow">video sectionUse JSON to describe your sitemaps. Don't waste your time with XML.Compress sitemaps with gzipInstallationpipinstallsitemapa# import in your scriptfromsitemapaimportSitemap,IndexSitemapUsageRead about usage on my website.Contactsvisit abstractkitchen.comdrop me a line:dmitry@abstractkitchen.comconnect with me on redditLicenseMIT
Package: sitemapbuilder
A simple sitemap builderThe sitemap builder traverses links from a website and constrains itself to
the given domain name. The final result will be a simple sitemap deduced
from the links visited. The crawler will accept & process only URLs with
http or https schemes.Installation and usageTo run the following command to install the tool:pipinstall-UsitemapbuilderTo run the sitemap builder:sitemapbuilder-u'https://monzo.com'-otest_monzo.dotSome websites have strong protection and the tool will not work for them:sitemapbuilder-u'https://bloomberg.com'-otest_bloomberg.dotHighlightsGenerate Graphviz.dotfile showing directed links between pages. One can generate PNG/PDF and other image/document formats.Haveconfigurable decay(maximum depth) to avoid abuse.Visit web link within the same hostname by default.Use5 threadsby default and times out after10 seconds.Timeout after5 secondswhen fetching a URL.Handle timeout exceptions when querying a website.Send aHTTP HEADrequest and verify thatContent-Typeistext/htmlandcharsetis eitherUTF-8orUS-ASCII.Have a map of visited URLs to avoid revisiting them.Follow HTTP redirects.Upcoming featuresConfigure the number of threads and timeout via cmd args.Allow web links from all subdomains.Allow web links from a list of domains.Allow web links matching a pattern.Add an option for hierarchical sitemap instead of directed graph.Use PriorityQueue instead of Queue to process links with higher decay first.Fine-graned info, warn and error logging.Pass seed links from a file.Save to and resume from a DB/persistent data source.Faster concurrency and better performance with asyncio.
Package: sitemap-builder
Sitemap BuilderGenerate sitemaps from programmatically collected URLs.FeaturesGenerates an index that points to compressed sitemaps fileseach generated sitemap won't have more than the allowed amount of
50.000 URLsjinja2 templatesInstallpip install sitemap-builderUsageImportfrom sitemap import SitemapInitializesitemap = Sitemap()Add urls to process them later:sitemap.add_url("dummy-page")
sitemap.add_url("another-page")
sitemap.add_url("category/foo-bar")Generate sitemap index and filessitemap.generate()Example:Generated files:.
├── sitemap.xml
└── sitemaps
    ├── sitemap-0.xml.gz
    ├── sitemap-1.xml.gz
    ├── sitemap-2.xml.gz
	└── sitemap-3.xml.gz
Package: sitemapcrawler
Sitemap Crawlersitemapcrawler is a simple, blocking Python Crawler that is the backbone of a few other projects.You're welcome to use it, but it's only as modular as we've needed it to be, which is to say, probably not fit for projects that aren't built with this in mind.It works pretty simply.Installationpip install sitemapcrawlerUsagefrom sitemapcrawler import Crawler
crawler = Crawler(domain="https://yourdomain.com", sitemap="https://yourdomain.com/sitemap.xml", fetch=True)
crawler.run()If you just want to fetch a given page, create an instance of the crawler and call it like this:crawler.fetch_page(url="https://yourdomain.com/blog/title")Theinitwill create a nanoidcrawl_idso that when results are persisted, they'll be associated to a given crawl, to make it easy for reports to be built against crawls and such.Building / Distributingpython3 -m build
python3 -m twine upload dist/* --skip-existing
Package: sitemap-generator
Sitemap generatorinstallingpip install sitemap-generatorrequirementsasyncio
aiofile
aiohttpexampleimport sys
import logging
from pysitemap import crawler
from pysitemap.parsers.lxml_parser import Parser

if __name__ == '__main__':
    if '--iocp' in sys.argv:
        from asyncio import events, windows_events
        sys.argv.remove('--iocp')
        logging.info('using iocp')
        el = windows_events.ProactorEventLoop()
        events.set_event_loop(el)

    # root_url = sys.argv[1]
    root_url = 'https://www.haikson.com'
    crawler(
        root_url, out_file='debug/sitemap.xml', exclude_urls=[".pdf", ".jpg", ".zip"],
        http_request_options={"ssl": False}, parser=Parser
    )TODObig sites with count of pages more then 100K will use more then 100MB
memory. Move queue and done lists into database. Write Queue and Done
backend classes based onListsSQLite databaseRedisWrite api for extending by user backendschangelog
Package: sitemap-maker
Python tool to generate sitemap XML files.This tool relies mainly on the Python crawlersitecrawl.InstallationUsing pip:pip3installsitemap_makersitemap_maker--helpOr build from sources:# Clone projectgitclonehttps://github.com/gabfl/sitemap_maker&&cdsitemap_maker# Installationpip3install.Usagesitemap_maker--urlhttps://www.weather.gov/\--sitemapoutput.xml\--depth1\--max5\--no_pound\--verbose# For help:
# sitemap_maker --helpWill create a filesitemap.xml:<?xml version="1.0" encoding="UTF-8"?><urlsetxmlns="http://www.sitemaps.org/schemas/sitemap/0.9"><url><loc>https://www.weather.gov</loc></url><url><loc>https://www.weather.gov/wrh/climate</loc></url><url><loc>https://www.weather.gov/safety/flood</loc></url><url><loc>https://www.weather.gov/safety/tsunami</loc></url><url><loc>https://www.weather.gov/safety/beachhazards</loc></url></urlset>
Package: site-map-parser
Site Map ParserScript and library which reads urls and converts to objects, allows exporting as CSV or JSON.Handle sitemaps according to:https://www.sitemaps.org/protocol.htmlInstallationpip install site-map-parserUsageScript usagesmapper $url > /tmp/data.csvLogs written to ~/sitemap_run.logArgumentsArgumentOptionsDefaultInformation-hN/AN/AOutputs argument dataurle.g.http://www.example.com-http://www.example.com/other_sitemap.xmlN/ARequired - sitemap data to retrieve-l, --logCRITICALorERRORorWARNINGorINFOorDEBUGINFOlogs to sitemapper_run.log in install folder-e, --exportercsvorjsoncsvExport format of the dataLibrary UsagefromsitemapparserimportSiteMapParsersm=SiteMapParser('http://www.example.com')# reads /sitemap.xmlifsm.has_sitemaps():sitemaps=sm.get_sitemaps()# returns iterator of sitemapper.Sitemap instanceselse:urls=sm.get_urls()# returns iterator of sitemapper.Url instancesExportingTwo exporters are available: csv and jsonCSV Exporterfromsitemapparser.exportersimportCSVExporter# sm set as per earlier library usage examplecsv_exporter=CSVExporter(sm)ifsm.has_sitemaps():print(csv_exporter.export_sitemaps())elifsm.has_urls():print(csv_exporter.export_urls())JSON Exporterfromsitemapparser.exportersimportJSONExporter# sm set as per earlier library usage examplejson_exporter=JSONExporter(sm)ifsm.has_sitemaps():print(json_exporter.export_sitemaps())elifsm.has_urls():print(json_exporter.export_urls())
Package: sitemapper
Permission is hereby granted, free of charge, to any person obtaining a copyof this software and associated documentation files (the "Software"), to dealin the Software without restriction, including without limitation the rightsto use, copy, modify, merge, publish, distribute, sublicense, and/or sellcopies of the Software, and to permit persons to whom the Software isfurnished to do so, subject to the following conditions:The above copyright notice and this permission notice shall be included in allcopies or substantial portions of the Software.THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS ORIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THEAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHERLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THESOFTWARE.Description: # SitemapperA utility for generating sitemaps from a list of URLs.## Installation```pip install sitemapper```## Usage```sitemapper --input-file URLs.txt --output-directory out --change-freq weekly --max-urls-per-sitemap 50000 --sitemap-root https://example.com --pretty-print```## Sample output```<?xml version='1.0' encoding='utf-8'?><sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"><sitemap><loc>https://marcomontagna.com/sitemap0.xml</loc><lastmod>2018-06-01T19:02:17.440168</lastmod></sitemap><sitemap><loc>https://marcomontagna.com/sitemap1.xml</loc><lastmod>2018-06-01T19:02:17.440168</lastmod></sitemap></sitemapindex>``````<?xml version='1.0' encoding='utf-8'?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"><url><loc>https://marcomontagna.com/</loc><changefreq>weekly</changefreq></url><url><loc>https://marcomontagna.com/blog.html</loc><changefreq>weekly</changefreq></url></urlset>```Keywords: sitemaps,sitemap index,sitemap generatorPlatform: UNKNOWNClassifier: Development Status :: 5 - Production/StableClassifier: Intended Audience :: DevelopersClassifier: Natural Language :: EnglishClassifier: License :: OSI Approved :: MIT LicenseClassifier: Programming Language :: PythonClassifier: Programming Language :: Python :: 2Classifier: Programming Language :: Python :: 2.7Classifier: Programming Language :: Python :: 3Classifier: Programming Language :: Python :: 3.6Requires-Python: >=2.7Description-Content-Type: text/markdown
Package: sitemap-preloader
Website SiteMap Preloader :rocket:Package for preloading website URLS from a provided SITEMAP URL.
You can use this package to preload your website URLS in the background, so that your website will be faster for your users.
You can set the depth in case the sitemap has more than one level.Requirements :clipboard:Python 3.8+Installation :wrench:Use the package managerpipto install preloader.pip install sitemap-preloaderHow to use :question:Follow the steps below to use the package on a python project.from preloader import Preloader
    sitemap_url = 'https://www.example.com/sitemap.xml'
    preloader = Preloader(sitemap_url, depth=2)

    preloader.fetch_pages()Testing :white_check_mark:We use pytest for testing. To run the tests, run the following command in the root directory of the project.pytest -sContributing :handshake:Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.
Package: sitemap_python
A Python utility for building sitemaps.UsageGenerate sitemapimport datetime
import sitemap.generator as generator

sitemap = generator.Sitemap()
sitemap.add("http://www.example.com",
            lastmod=datetime.datetime.now(),
            changefreq="monthly",
            priority="1.0")
sitemap_xml = sitemap.generate()


sitemap_index = generator.Sitemap(type='sitemapindex')
sitemap_index.add("http://www.example.com/sitemap01.xml",
                  lastmod=datetime.datetime.now(),
sitemap_index_xml = sitemap_index.generate()Ping search engineCurrently support ping Google and Bing with sitemap urls.import sitemap.ping as ping

ping.ping("google", "http://www.example.com/sitemap.xml")
ping.ping_urls("bing", ["http://www.example.com/sitemap.xml"])Push url to BaiduPush urls directly to Baidu. Related document availableat
here.import sitemap.baidu as baidu
bp = baidu.BaiduPush("http://www.example.com", "<YOUR_KEY>")
bp.add("http://www.example.com/example.html")
bp.flush()Verify the spider ip addresssitemap.spidercan be use to verify whether the ip address of spider
is genius.Example:from sitemap.spider import get_verified_spider_name

# spider_name will be None if no search engine is matched
spider_name = get_verified_spider_name("66.249.65.219")The methodget_verified_spider_namehas usessocket.gethostbyaddr, which may be slow in some cases. So makeguess_spider_name_from_uamethod may filter out several results
via User-Agent.from sitemap.spider import get_verified_spider_name, guess_spider_name_from_ua

spider_name = guess_spider_name_from_ua(spider_ua)
if spider_name:
    spider_name = get_verified_spider_name(spider_ip)
Package: sitemap-range-fetch
AboutThis module provides theSitemapRangeclass and a tool to allow command-line usagesitemap_fetch.py.The classSitemapRangeis meant primarily as a generic building block for creating news aggregating applications where the datasources
arespec-compliantnews websites.There are some fault-tolerance features included to deal with some inconsistencies in sitemaps.InstallTo install from pypi:pip3 install --user sitemap-range-fetchUsageFetching all news articles oncnn.comin the past 6 days, and format the result asJSON:sitemap_fetch.py --site "https://cnn.com" --format json --daysago 6Here is an example of using theSitemapRangeclass in your code:from sitemap_range.sitemap_range import SitemapRange
from datetime import datetime, timedelta
sr = SitemapRange("https://cnn.com")
in_range = sr.get_articles_in_range(start=datetime.now()-timedelta(days=3), end=datetime.now(), opts={})
print(in_range)Theget_articles_in_rangemethod returns a list of dictionaries, where each dictionary has two
keys:"url"and"dt"which is anISO 8601 formatted datetime string(as returned by theisoformat method).More details about the CLI switches:usage: sitemap_fetch.py [-h] --site SITE [--format FORMAT] [--daysago DAYSAGO]
                            [--notz] [--advanced]

    Tool for extracting articles from news websites

    optional arguments:
      -h, --help         show this help message and exit
      --site SITE        the url for the website
      --format FORMAT    the url for the website
      --daysago DAYSAGO  defines the oldest date of an article that will be
                         selected (default: 2 days ago)
      --notz             strip the timezone from the dates before selection
                         (processing is more fault-tolerant)
      --advanced         use a more fault-tolerant parserDetailsThis module is provided as is underMIT License.For extensions, customizations or business inquiries you canget in touch here.