Saravanan666/pypi-org huggingface dataset:
Package: web-archive-get
web_archive_geta tool to find archived web pages from different websites using multiple different services. the currently three functions implemented and that are:list_pageget a pages of the urlurl -  the url of the page
RETURN a list of requestable objectlist_subdoaminget all subdoamin of a domain your looking upurl -  the url of the domain
RETURN a list of requestable objectsearch_url_hostlook up website with the same hosturl -  the url of the host
RETURN a list of requestable objectsearch_url_subpathget the all subpath  of the urlurl -  the url of the subpath
RETURN a list of requestable objectrequestable objectrequest()get_length()get_statuscode()get_mimetype()get_url()
Package: webarchiver
WebarchiverVersion: 0.32.4Python tool that allows you to take full page screenshots of pages without adsSupports batching by adding multiple links in a text file, or my adding links to command line separated by commas.This repository is actively maintained - Contributions are welcome!Requirements:One of the following browsers:Chrome/Chromium browserFirefoxSelenoid ServerUsage:Short FlagLong FlagDescription-h--helpSee Usage-b--browserSpecify browser: Chrome / Firefox / Selenoid-c--cleanConvert mobile sites to regular site-d--directoryLocation where the images will be saved--dpiDPI for the image-e--executorExecution environment: Local / Selenoid Host|Selenoid URL-f--fileText file to read the URL(s) from-l--linksComma separated URL(s)-i--image-typeSave images as PNG or JPEG-p--processesNumber of processes to run concurrently-s--scrapeScrape URL(s) by Downloading-u--url-filterFilter URL(s) that contain this string-z--zoomThe zoom to use on the browserExample:webarchiver-c-f<links_file.txt>-l"<URL1,URL2,URL3>"-i'jpeg'-d"~/Downloads"-z100--dpi1--browser"Firefox"webarchiver-c-f<links_file.txt>-l"<URL1,URL2,URL3>"-i'png'-d"~/Downloads"-z100--dpi1--executor"selenoid|http://selenoid.com/wd/hub"--browser"Chrome"webarchiver-s-f<links_file.txt>-l"<URL1,URL2,URL3>"Installation Instructions:Install Python Packagepython-mpipinstallwebarchiverGeniusbot ApplicationUse with a GUI through GeniusbotVisit ourGitHubfor more informationInstallation Instructions with Geniusbot:Install Python Packagepython-mpipinstallgeniusbotRepository Owners:
Package: webargs
Homepage:https://webargs.readthedocs.io/webargs is a Python library for parsing and validating HTTP request objects, with built-in support for popular web frameworks, including Flask, Django, Bottle, Tornado, Pyramid, Falcon, and aiohttp.fromflaskimportFlaskfromwebargsimportfieldsfromwebargs.flaskparserimportuse_argsapp=Flask(__name__)@app.route("/")@use_args({"name":fields.Str(required=True)},location="query")defindex(args):return"Hello "+args["name"]if__name__=="__main__":app.run()# curl http://localhost:5000/\?name\='World'# Hello WorldInstallpip install -U webargswebargs supports Python >= 3.8.DocumentationFull documentation is available athttps://webargs.readthedocs.io/.Support webargswebargs is maintained by a group ofvolunteers.
If you’d like to support the future of the project, please consider
contributing to our Open Collective:Professional SupportProfessionally-supported webargs is available through theTidelift Subscription.Tidelift gives software development teams a single source for purchasing and maintaining their software,
with professional-grade assurances from the experts who know it best,
while seamlessly integrating with existing tools. [Get professional support]Security Contact InformationTo report a security vulnerability, please use theTidelift security contact.
Tidelift will coordinate the fix and disclosure.Project LinksDocs:https://webargs.readthedocs.io/Changelog:https://webargs.readthedocs.io/en/latest/changelog.htmlContributing Guidelines:https://webargs.readthedocs.io/en/latest/contributing.htmlPyPI:https://pypi.python.org/pypi/webargsIssues:https://github.com/marshmallow-code/webargs/issuesEcosystem / related packages:https://github.com/marshmallow-code/webargs/wiki/EcosystemLicenseMIT licensed. See theLICENSEfile for more details.
Package: webargs-quixote
webargs-quixotewebargssupport for Quixote web framework.Installpipinstallwebargs-quixote# orpoetryaddwebargs-quixoteUsagefromwebargsimportfieldsfromwebargs_quixoteimportparser,use_args@use_args({"value":fields.Int()})defhome(req,args):returnargsLookswebargsdocs for more details.
Package: web-article
web_article: A Python Article Scraper and CLI Toolweb_articleis a Python package designed to scrape and parse articles from various web sources seamlessly. Accompanied by a command-line interface (CLI) tool, users can efficiently extract articles from given URLs and export them in multiple formats.FeaturesScrape articles from multiple supported URLs.Export articles to various formats: JSON, JL, and CSV.Intuitive command-line tool for quick extraction and export.Flexible output options, including encoding, indentation, and delimiters.RequirementsPython 3.xInstallationpython3-mpipinstallweb_articleUsageUsing the web_article CLI tool is straightforward. Here are some common use cases:Scraping a Single Article from URL:web_articleurl1Scraping Multiple Articles from URLs:web_article url1 url2 url3Scraping Articles from URLs Listed in a File:web_article urls.txtExporting Articles in JSON Format:web_article urls.txt -o out.json -f jsonExporting Articles in JSON Format with Indentation:web_article urls.txt -o out.json -f json -i 2Exporting Articles in CSV Format:web_article urls.txt -o out.csv -f csvExporting in a Specific Encoding (e.g., UTF-8 with BOM for Office Excel):web_article urls.txt -o out.csv -f csv -e utf-8-sigOptions:-o, --outfile : The name of the output file. Defaults to standard output if not provided.

-f, --outfmt : The format for the output. Choices are 'jl', 'json', 'csv'. Defaults to 'jl'.

-d, --delimiter : Delimiter for CSV output. Default is ','.

-e, --encoding : Encoding for the output file. Default is 'utf-8'.

-i, --indent : Indentation for JSON output. An integer value.
Package: webarticle2text
# Webarticle2Text - Extracts the main article text from a webpage.[![](https://img.shields.io/pypi/v/webarticle2text.svg)](https://pypi.python.org/pypi/webarticle2text) [![Build Status](https://img.shields.io/travis/chrisspen/webarticle2text.svg?branch=master)](https://travis-ci.org/chrisspen/webarticle2text) [![](https://pyup.io/repos/github/chrisspen/webarticle2text/shield.svg)](https://pyup.io/repos/github/chrisspen/webarticle2text)## OverviewThis project is obsolete and now only serves as a reference. I recommend you instead use [newspaper](https://github.com/codelucas/newspaper), which is an order-of-magnitude more accurate than any other article extraction library I’ve encountered.Please seecompare.csvfor a performance comparison of several similar tools.This attempts to locate and extract the largest cluster of text in a
webpage. It does this by walking the DOM-tree, identifying all text
segments and their depth inside the DOM, appends all text at roughly
the same depth, and then returns the chunk with the largest total
length.This approach usually works well with typical news sites where one
news article is displayed per URL. This approach usually fails with
URLs displaying multiple news blurbs (e.g. news aggregators).## InstallationYou may need to install the tidylib system package, which you can get on Ubuntu 12.04 using:sudo apt-get install libtidy-0.99-0or on Fedora using:sudo yum install libtidyThen, simply install the package using pip:pip install webarticle2text## UsageYou can invoke the script either as a Python module:from webarticle2text import webarticle2text
print webarticle2text.extractFromURL(”http://some/arbitrary/url”)or as a standalone command line script:webarticle2text.pyhttp://some/arbitrary/urlNote, to use it from the command line, you’ll need to ensure it has execute
permission and is located in your PATH. On most platforms, this should
automatically be done by setup.py.## DevelopmentTests require the Python development headers to be installed, which you can install on Ubuntu with:sudo apt-get install python-dev python3-dev python3.4-devTo run unittests across multiple Python versions, install:sudo apt-get install python3.4-minimal python3.4-dev python3.5-minimal python3.5-devTo run all [tests](http://tox.readthedocs.org/en/latest/):export TESTNAME=; toxTo run tests for a specific environment (e.g. Python 2.7):export TESTNAME=; tox -e py27To run a specific test:export TESTNAME=.test_extract; tox -e py27## History1.0.0 (2008.9.16) Initial public release.1.2.0 (2011.1.3) Update to support Unicode.1.2.2 (2011.12.17) Cleaned up installation procedure and documentation and moved to github.com.1.2.3 (2011.12.21) Fixed encoding error when redirecting stdout. e.g. webarticle2text.pyhttp://some/arbitrary/url> output.txt1.2.5 (2012.11.5) Added the option to specify user-agent header to use when requesting URLs.2.0.0 (2014.4.20) Added support for Python 3.2.
Package: webassets-babel
# Webassets BabelBabel filter for [Webassets](https://webassets.readthedocs.org/en/latest/)## DescriptionTurn your ES6+ code into ES5 friendly code usinghttps://babeljs.io/## Installation` npm install-gbabel pip installwebassets-babel`## UsingFirst register filter:` from webassets.filter import register_filter from webassets_babel import BabelFilter register_filter(BabelFilter) `Then you can use it in your build process:`Bundle('file.js',filters='babel')`You can change babel binary using BABEL_BIN setting of your application.History0.3.0 (2015-10-09)Bug fix0.2.0 (2015-09-27)Enable filter in debug mode0.1.2 (2015-04-26)First release on PyPI.
Package: webassets-browserify
Filter for for compiling assets usingBrowserifyandwebassets.Basic usagefromwebassets.filterimportregister_filterfromwebassets_browserifyimportBrowserifyregister_filter(Browserify)Usage with DjangoThis requiresdjango-assets.fromdjango_assetsimportBundle,registerfromwebassets.filterimportregister_filterfromwebassets_browserifyimportBrowserifyregister_filter(Browserify)js=Bundle('js/main.js',filters='browserify',output='bundle.js',depends='js/**/*.js')register('js_all',js)
Package: webassets-closure-soy
UsageEasiest way to install:pip install closure-soypip install webassets-closure-soyYou can set environment variables to use your own SoyToJsSrcCompiler.jar
but this package will look for theclosure-soypackage for easy setup.In your assets.py file:from webassets_ext import ClosureSoyFilterfrom webassets.filter import register_filterregister_filter(ClosureSoyFilter)Then use filter=”closure_soy” wherever you want to use it.
If you want to chain it with a js minifer, make sure that
closure_soy comes first in the list of filters.BugsIf you have any issues, please open a ticket at theGithub page.
Package: webassets-elm
Elm filter for webassetsFilter for compilingElmfiles usingwebassets.Installpip install webassets-elmAs of version 0.2.0, this plugin requiresElm 0.19or newer (building withelm make).If you need to build your Elm project withelm-make(Elm 0.18 and older), you can pin yourwebassets-elmpackage to version0.1.7.Basic usagefromwebassets.filterimportregister_filterfromwebassets_elmimportElmregister_filter(Elm)SettingsOptionallyas an evironment variable you can have:ELM_BIN: alternative path toelmif it isnotavailable globally (e.g.node_modules/.bin/elm).ELM_OPTIMIZE: enable the Elm compiler optimization option. Recommended for production output.ELM_DEBUG: enable the Elm compiler debug option.ExamplesFlask withflask-assetsfromflaskimportFlaskfromflask_assetsimportBundle,Environmentfromwebassets.filterimportregister_filterfromwebassets_elmimportElmapp=Flask(__name__)register_filter(Elm)assets=Environment(app)elm_js=Bundle('elm/main.elm',filters=('elm',),output='app.js')assets.register('elm_js',elm_js)Django withdjango-assetsfromdjango_assetsimportBundle,registerfromwebassets.filterimportregister_filterfromwebassets_elmimportElmregister_filter(Elm)elm_js=Bundle('elm/main.elm',filters=('elm',),output='app.js')register('elm_js',elm_js)ContributingFeel free toreport an issue,open a pull request, ordrop a line.Don’t forget to write and run tests, and format code withBlack:python setup.py test
black .Please note you needelmbinary available to run tests, here you can find different ways toinstall Elm.
Package: webassets-iife
webassets-iifeis awebassetsfilter to wrap a JavaScript bundle in an
IIFE to prevent global leaks and improve minification.Installpip install webassets-iifeUsageFor example with Flask:from flask.ext.assets import Environment, Bundle
from webassets_iife import IIFE

# ...

assets = Environment(app)

js = Bundle('myscript1.js',
            'myscript2.js',
            filters=(IIFE, 'closure_js'), output='all.min.js')
assets.register('js_all', js)This will concatmyscript1.jsandmyscript2.jsinto one JS chunk, wrap
it in an IIFE and minify it.IIFE?AnIIFEis an Immediately-Invoked Function Expression. It’s an anonymous
function that’s declared and invoked immediately after that. It’s used in
JavaScript to create a closed environment which can’t be accessed from the
outside.// a and b can be accessed by external code
var a = 3,
    b = 1;
// ... some code ...

// a and b can't be accessed by external code
(function() {
    var a = 3,
        b = 1;
    // ... some code ...
})();Wrapping code in an IIFE helps the minifier see the dead code, because itknowsthat these local variables can’t be accessed from the outside and thus
can remove them or mangled their name.
Package: webassets-jinja2js
UsageEasiest way to install:pip install pwt.jinja2jspip install webassets-jinja2jsIn your assets.py file:from webassets_ext import JinjaToJSFilterfrom webassets.filter import register_filterregister_filter(JinjaToJSFilter)Then use filter=”jinja2js” wherever you want to use it.
If you want to chain it with a js minifer, make sure that
jinja2js comes first in the list of filters.BugsIf you have any issues, please open a ticket at theGithub page.
Package: webassets-libsass
webassets-libsass=================Filter for asset management "webassets" that uses "libsass"Required========libsass (libsass-python): http://dahlia.kr/libsass-python you need to have C/C++ compilerwebassets: https://pypi.python.org/pypi/webassetsInstallation============Option 1:========download and extract, then: python setup.py installOption 2:========run pip install webassets-libsassUse===Register filter:===============from webassets.filter import register_filterfrom webassets_libsass import LibSassregister_filter(LibSass)Like another webassets filter:=============================# foundation frameworkfoundation = Bundle('scss/foundation/app.scss',filters='libsass',output='css/foundation.css')Configuration Options=====================LIBSASS_OUTPUT_STYLE (output_style)an optional coding style of the compiled result. choose one of: nested (default), expanded, compact, compressedLIBSASS_INCLUDE_PATHS (include_paths)an optional list of paths to find @imported SASS/CSS source filesExample:env = Environment( ... )env.config['LIBSASS_OUTPUT_STYLE'] = 'compressed'Bundle('src/test.scss', filters='libsass', output='out/test.css')
Package: webassets_ng_annotate
Webassets-ng-annotate=====================Ng-annotate filter for webassets.Install-------npm install -g ng-annotatepip install webassets-ng-annotateUsage-----from webassets import Bundleimport webassets_ng_annotatewebassets_ng_annotate.register()js = Bundle('vendors/angular.js', 'app.js',filters='ng-annotate,uglifyjs',output='gen/packed.js')Parameters----------- `NG_ANNOTATE_BIN` - path to `ng-annotate` binary- `NG_ANNOTATE_EXTRA_ARGS` - additional option to pass to `ng-annotate`
Package: webassets-recess
Install:pip install webassets-recessRegister:from webassets.filter import register_filter
from webassets_recess import RecessFilter

register_filter(RecessFilter)Example usage:from webassets.loaders import YAMLLoader
for name, bundle in YAMLLoader('webassets.yaml').load_bundles().items():
    assets.register(name, bundle)Where webassets.yaml contains:css_all:
filters: recess,cssutils
output: cache/default.css
contents:
    - bootstrap/less/bootstrap.lessNow you can hack bootsrap less. Merges are on you.
Package: webassets-webpack
Filter for for compiling assets usingWebpackandwebassets.Basic usagefromwebassets.filterimportregister_filterfromwebassets_browserifyimportWebpackregister_filter(Webpack)Usage with DjangoThis requiresdjango-assets.fromdjango_assetsimportBundle,registerfromwebassets.filterimportregister_filterfromwebassets_webpackimportWebpackregister_filter(Webpack)js=Bundle('js/main.js',filters='webpack',output='bundle.js',depends='js/**/*.js')register('js_all',js)
Package: web-asset-vendor
web-asset-vendorDownloading assets from the web in a reproducible way.Intentionally this was used to fetch web-dev assets like .js ans .css files from cdn's but the generic resolver pattern allows you to download everything you want (images, binarys, ...).Example Usagemain.pyfromweb_asset_vendorimportFetcherfetcher=Fetcher.from_yaml("dependencies.yaml")fetcher.fetch()Note: Optionally one will not simply initialize Fetcher with a yaml file. Instead, one would use the Constructor.dependencies.yamlconfig:output:vendor/fixiation:True# WIPresolvers:# CLOUDFLARE-url:https://cdnjs.cloudflare.com/ajax/libs/${author}/${version}/js/${package}.min.js-url:https://cdnjs.cloudflare.com/ajax/libs/${author}/${version}/css/${package}.min.csstags:[css]# UNPKG-url:https://unpkg.com/${package}@${version}/dist/${package}.min.js-url:https://unpkg.com/${package}@${version}/dist/css/${package}.min.csstags:[css]assets:bootstrap.min.css:folder:css/bootstrapresolveWith:[css]resolvedBy:author:twitter-bootstrapversion:4.6.0package:bootstrapbootstrap.min.js:folder:js/bootstrapresolvedBy:author:twitter-bootstrapversion:4.6.0package:bootstrapjquery.min.js:resolvedBy:version:3.6.0package:jqueryexample output:.
+-- vendor
|   +-- css
|   |   +-- bootstrap
|   |   |   +-- bootstrap.min.css
|   +-- js
|   |   +-- bootstrap
|   |   |   +-- bootstrap.min.js
|   +-- uncategorized
|   |   +-- jquery.min.css
Package: web-automation
Web Automation FrameworkThis is a project designed to minimize the setup and config of selenium and actions performed on elementsTo use this library, create a snippet like below and populate the actions in therunmethod (remove thepasskeyword)fromweb_automation.frameworkimportFramework,Browser,ByclassCustomClass(Framework):defrun(self):passCustomClass(browser=Browser.CHROME,wait=10,headless=False,download_path='')The Constructor accepts 4 params (only one mandatory: browser) which is applied across the automation sessionbrowser: Browser- The type of browser that you want to use, based on theBrowserclasswait: int- The wait time for an element before throwingNoSuchElementExceptionexception, default value is30secondsheadless: bool- Boolean option to set the automation to use the browser in headless mode (runs in background and no visible browser window), default value isFalsedownload-path: str- the path to store the downloaded files, default value is the system download folderBrowser ClassTheBrowserenum from the module will provide the target browser required to run the automation.The supported browsers areGoogle ChromeMicrosoft EdgeMozilla FirefoxSafariElement selectionThe HTML elements in the browser can be selected using one of the below attributesidnametag nameclass namecss selectorxpathlink textpartial link textThese selectors are provided using theByclass (Selenium Class)SelectBy ClassThere are different ways to select an option in the dropdown menu. TheSelectByenum has the below listed options to select the item in the dropdownIndexTextValueActions AvailableThe below methods are available in the Framework Class and should be used inside the run method prefixingself.type(elem_type, elem_id, value, clear)- used to type thevaluein the elementelem_type: By- one of the options fromByelem_id: str- element ID value based on theelem_typevalue: str- value to be typed in the componentclear: bool- Boolean value to clear the element before typingclick(elem_type, elem_id)- used to click the elementelem_type: By- one fo the options fromByelem_id: str- element ID value based on theelem_typeselect(elem_type, elem_id, select_by, value)- used to select an options from the dropdown inputelem_type: By- one of the options fromByelem_id: str- element ID value based on theelem_typeselect_by: SelectBy- one of the options fromSelectByvalue: str- value for theSelectByoption typedeselect(elem_type, elem_id)- used to deselect a dropdown listelem_type: By- one of the options fromByelem_id: str- element ID value based on theelem_typecheck_if_exists(elem_type, elem_id)- used to check if the mentioned element exists in the pageelem_type: By- one of the options fromByelem_id: str- element ID value based on theelem_typenavigate_to(url)- used to navigate to the mentioned urlurl: str- Target url to be navigated towait(wait_time)- wait for mentioned timewait_time: int- wait for the mentioned number of seconds
Package: webb
## Webb - A Complete Web Scrapper and Crawler Library
An all-in-one Python library to scrap, parse and crawl web pages### Gist
This is a light-weight, dynamic and highly-flexible Python library. It can be used tocrawl, download, index, parse, scrap and analyze web pagesin a systematic manner or any of the individual functionality. It is also used toclean web pages, normalize web pages, store web data, extract server-side informationandimport/export relevant componentsfrom the web. Some of the other features also includedownloading images from a web page, downloading google images and spidering wikipedia articles.### Usage and Instructions
For usage and instructions please visit the [Official Documentation](https://github.com/hardikvasa/webb/blob/master/docs/Documentation.md)For issues and discussion visit the [Issue Tracker](https://github.com/hardikvasa/webb/issues)For sample codes and examples, please visit [Examples Codes](https://github.com/hardikvasa/webb/tree/master/examples)### Compatability
This library is compatible with both Python 2 (2.x) as well as Python 3 (3.x) versions. It is a download-import-and-run program with no or little changes as required by users.### Dependencies
There areno dependenciesto this project. Hurray! It functions entirely of the standard ‘built-in’ library support. It does not need any external support or installations. Just download and run!!!### Status
This is a stand-alone python script which is ready-to-run, but still under development. Many more features will be added to it shortly.### Disclaimer
The crawler function lets you download  and crawl tons of web pages. Please do not download and crawl any pages of a domain without reading the ‘robot.txt’ file of that specific domain.It is inappropriate to violate the robot.txt file and is strictly not recommended. This may even lead to the domain completely blocking your crawler and thus blacklisting it. It is also not appropriate to crawl pages at high rate as it may put a lot of pressure on the requesting server.