Saravanan666/pypi-org huggingface dataset:
Package: scrapist
Scrapist: The Next Level of Efficient Web ScrapingScrapist is a web scraper designed for Python. This web scraper uses requests and BeautifulSoup and also provides support for Scrapy style CSS selectors. Its features are:Faster than requests and BeautifulSoup.Is effective in fetching multiple pages compared to Scrapy.Provides support for both BeautifulSoup-style selection and Scrapy-style CSS selection.InstallationTo install Scrapist, run this command in the terminal:pip install scrapistInitializationTo start web scraping with Scrapist, use this code:fromscrapistimportScraperscraper=Scraper()data=scraper.scrape("<your url here>")print(data.soup)Getting Specified Parts/Tags of a web pageTo get specified parts/tags of a web page, you can choose either of the two ways:The Scrapy-styleTo get specified data Scrapy-style, use this code after the initialization:first=data.css("<your css selector here>").get()print(first)# Orall_data=data.css("<your css selector here>").getall()print(all_data)The BeautifulSoup-styleTo get specified data BeautifulSoup-style, use this code after the initialization:first=data.find("<your tag here>","[your attributes here]")print(first)# Orall_data=data.find_all("<your tag here>","[your attributes here]")print(all_data)Creating a Soup StrainerTo create a soup strainer, use this code just after the line of creating a scraper (Thescraper = Scraper()intialization line):strainer=scraper.strainer("<your tag here>","[your attribute here]")And use the strainer in thestrainerparameter inscraper.scrape()function.
Package: scraple
ScrapleScraple is a Python library designed to simplify the process of web scraping,
providing easy scraping and easy searching for selectors.Versionv0.1.1changelogInstallationThe package is hosted inPypiand can be
installed using pip:pipinstallscrapleMain APIThe package provides two main classes: Rules and SimpleExtractor.1. RulesThe Rules class allows you to define rules of extraction.
You can pick selector just by knowing what string present in that page using theadd_field_rulemethod.
This method automatically searches for selector of element which text content match the string.
Additionally, theadd_field_rulemethod supports regular expression matching.fromscrapleimportRules#To instantiate Rules object you need to have the reference page.some_rules=Rules("reference in the form of string path to local html file","local")some_rules.add_field_rule("a sentence or word exist in reference page","field name 1")some_rules.add_field_rule("some othe.*?text","field name 2",re_flag=True)# Add more field rules...# It automatically search for the selector, to see it you can see the rule in console# or by printing it# print(rules)2. SimpleExtractorThe SimpleExtractor class performs the actual scraping based on a defined rule.
A Rules object act as the "which to extract" and the SimpleExtractor do the "extract" or
scraping. First, pass a Rules object
to SimpleExtractor constructor and use theperform_extractionmethod to create a generator object that iterate dictionary of
elements extracted.fromscrapleimportSimpleExtractorextractor=SimpleExtractor(some_rules)# some_rules from above code snippetresult=extractor.perform_extraction("web page in the form of beautifulSoup4 object","parsed")# print(next(result))# {#   "field name 1": [element, ...],#   "field name 2": ...,#   ...# }For more information and tutorial, see thedocumentationor
visit the mainrepository
Package: scrapli
scrapliDocumentation:https://carlmontanari.github.io/scrapliSource Code:https://github.com/carlmontanari/scrapliExamples:https://github.com/carlmontanari/scrapli/tree/master/examplesscrapli -- scrap(e c)li --  is a python 3.7+ library focused on connecting to devices, specifically network devices
(routers/switches/firewalls/etc.) via Telnet or SSH.Key Features:Easy: It's easy to get going with scrapli -- check out the documentation and example links above, and you'll be
connecting to devices in no time.Fast: Do you like to go fast? Of course you do! All of scrapli is built with speed in mind, but if you really
feel the need for speed, check out thessh2transport plugin to take it to the next level!Great Developer Experience: scrapli has great editor support thanks to being fully typed; that plus thorough
docs make developing with scrapli a breeze.Well Tested: Perhaps out of paranoia, but regardless of the reason, scrapli has lots of tests! Unit tests
cover the basics, regularly ran functional tests connect to virtual routers to ensure that everything works IRL!Pluggable: scrapli provides a pluggable transport system -- don't like the currently available transports,
simply extend the base classes and add your own! Need additional device support? Create a simple "platform" inscrapli_communityto easily add new device support!But wait, there's more!: Have NETCONF devices in your environment, but love the speed and simplicity of
scrapli? You're in luck! Check outscrapli_netconf!Concurrency on Easy Mode:Nornir'sscrapli plugingives you all the normal benefits of scrapliplusall the great features of Nornir.Sounds great, but I am a Gopher: For our Go loving friends out there, check outscrapligofor a similar experience, but in Go!RequirementsMacOS or *nix1, Python 3.7+scrapli "core" has no requirements other than the Python standard library2.1Although many parts of scraplidorun on Windows, Windows is not officially supported2While Python 3.6 has been dropped, itprobablystill works, but requires thedataclassbackport as well as third partyasync_generatorlibrary, Python 3.7+ has no external dependencies for scrapli "core"Installationpip install scrapliSee thedocsfor other installation methods/details.A Simple ExamplefromscrapliimportScraplidevice={"host":"172.18.0.11","auth_username":"scrapli","auth_password":"scrapli","auth_strict_key":False,"platform":"cisco_iosxe"}conn=Scrapli(**device)conn.open()print(conn.get_prompt())* Bunny artwork by Caroline Montanari, inspired by@egonelbre.
The bunny/rabbit is a nod to/inspired by the white rabbit inMonty Python and the Holy Grail, because there
are enough snake logos already!
Package: scrapli-cfg
scrapli_cfgDocumentation:https://scrapli.github.io/scrapli_cfgSource Code:https://github.com/scrapli/scrapli_cfgExamples:https://github.com/scrapli/scrapli_cfg/tree/master/examplesscrapli_cfg makes merging or replacing device configurations over Telnet or SSH easy, all while giving you the
scrapli behaviour you know and love.Key Features:Easy: It's easy to get going with scrapli and scrapli-cfg -- check out the documentation and example links above,
and you'll be managing device configurations in no time.Fast: Do you like to go fast? Of course you do! All of scrapli is built with speed in mind, but if you really
feel the need for speed, check out thessh2transport plugin to take it to the next level! All the "normal"
scrapli transport plugin goodness exists here in scrapli-cfg too!Great Developer Experience: scrapli_cfg has great editor support thanks to being fully typed; that plus
thorough docs make developing with scrapli a breeze.RequirementsMacOS or *nix1, Python 3.7+scrapli_cfg's only requirements isscrapli.1Although many parts of scraplidorun on Windows, Windows is not officially supportedInstallationpip install scrapli-cfgSee thedocsfor other installation methods/details.A simple ExamplefromscrapliimportScraplifromscrapli_cfgimportScrapliCfgdevice={"host":"172.18.0.11","auth_username":"scrapli","auth_password":"scrapli","auth_strict_key":False,"platform":"cisco_iosxe"}withopen("myconfig","r")asf:my_config=f.read()withScrapli(**device)asconn:cfg_conn=ScrapliCfg(conn=conn)cfg_conn.prepare()cfg_conn.load_config(config=my_config,replace=True)diff=cfg_conn.diff_config()print(diff.side_by_side_diff)cfg_conn.commit_config()cfg_conn.cleanup()
Package: scrapli-community
scrapli_communityDocumentation:https://scrapli.github.io/scrapli_communitySource Code:https://github.com/scrapli/scrapli_communityThis is the scrapli_community repository forscrapli.If you would like to use scrapli, but the platform(s) that you work with are not supported in the "core" scrapli
platforms, you have come to the right place! This library is intended to be a place for scrapli users to add
additional platform support to scrapli.Please see the mainscrapli repofor much more information about the main
project.Key Features:Easy: It's easy to get going with scrapli -- check out the documentation and example links above, and you'll be
connecting to devices in no time.Fast: Do you like to go fast? Of course you do! All of scrapli is built with speed in mind, but if you really
feel the need for speed, check out thessh2transport plugin to take it to the next level!Great Developer Experience: scrapli has great editor support thanks to being fully typed; that plus thorough
docs make developing with scrapli a breeze.Well Tested: Perhaps out of paranoia, but regardless of the reason, scrapli has lots of tests! Unit tests
cover the basics, regularly ran functional tests connect to virtual routers to ensure that everything works IRL!Pluggable: scrapli provides a pluggable transport system -- don't like the currently available transports,
simply extend the base classes and add your own! Need additional device support? Create a simple "platform" inscrapli_community -- this repo!to easily add new device support!Concurrency on Easy Mode:Nornir'sscrapli plugingives you all the normal benefits of scrapliplusall the great features of Nornir.RequirementsMacOS or *nix1, Python 3.7+1Although many parts of scraplidorun on Windows, Windows is not officially supportedInstallationpip install scrapli-communitySee thedocsfor other installation methods/details.A simple ExamplefromscrapliimportScraplimy_device={"host":"172.18.0.11","auth_username":"scrapli","auth_password":"scrapli","auth_strict_key":False,"platform":"ruckus_fastiron"}conn=Scrapli(**my_device)conn.open()
Package: scrapli-ipython
scrapli_ipythonscrapli_ipythonworks as an IPython magic and connect to any network devices withscrapli.In[1]: %pip install scrapli_ipython
       %load_ext scrapli_ipython

In[2]: %scrapli --platform cisco_iosxe 192.168.1.101

In[3]: %%cmd
       show interface description
       show ipv4 interface briefSee anotebook example
Package: scrapli-netconf
scrapli_netconfDocumentation:https://scrapli.github.io/scrapli_netconfSource Code:https://github.com/scrapli/scrapli_netconfExamples:https://github.com/scrapli/scrapli_netconf/tree/master/examplesscrapli_netconf is a NETCONF driver built on top of scrapli, giving you all the scrapli behaviour you know and love,
but for NETCONF connections.Key Features:Easy: Just like scrapli, scrapli_netconf is easy to get going with, and looks and feels just like "normal"
scrapli -- check out the documentation and example links above, and you'll be connecting to devices in no time.Fast: Do you like to go fast? Of course you do! scrapli_netconf supports all of the ssh transports that
scrapli core supports; check out thessh2transport if you've got the need for speed!Great Developer Experience: scrapli_netconf has great editor support thanks to being fully typed; that plus
thorough docs make developing with scrapli a breeze.Well Tested: Perhaps out of paranoia, but regardless of the reason, scrapli_netconf has lots of tests! Unit tests
cover the basics, regularly ran functional tests connect to virtual routers to ensure that everything works IRL!Concurrency on Easy Mode:Nornir'sscrapli plugingives you all the normal benefits of scrapliplusall the great features of Nornir.RequirementsMacOS or *nix1, Python 3.7+scrapli_netconf's only requirements arescrapli, of course, andlxml.1Although many parts of scraplidorun on Windows, Windows is not officially supportedInstallationpip install scrapli_netconfSee thedocsfor other installation methods/details.A Simple Examplefromscrapli_netconf.driverimportNetconfDrivermy_device={"host":"172.18.0.13","auth_username":"scrapli","auth_password":"scrapli","auth_strict_key":False,"port":830}conn=NetconfDriver(**my_device)conn.open()response=conn.get_config(source="running")print(response.result)
Package: scrapli-replay
scrapli_replayDocumentation:https://scrapli.github.io/scrapli_replaySource Code:https://github.com/scrapli/scrapli_replayExamples:https://github.com/scrapli/scrapli_replay/tree/main/examples/simple_test_casescrapli_replay: Tools to enable easy testing of scrapli programs and to create semi-interactive SSH servers that
look and feel like "real" network devices!Key Features:Easy: Easily test scrapli code with Pytest, or create mock SSH servers to play with!Pytest: Love scrapli and Pytest? Want to test your code that contains scrapli components, but can't test
against real devices in your CI? scrapli_replay is a Pytest plugin to help you with exactly this -- its likeVCR.pyandpytest-vcr, but for scrapli!Offline: Want to be able to have the look and feel of a network device without having a network device? Create
a mock SSH server based on real network device behavior and run it in your CI or a Raspberry Pi (or whatever)!Lightweight: Want to spin up a bunch of test servers, but don't have the resources for real device images?
Asyncssh is fast and lightweight so you can run loads of mock SSH servers with very little resources!Installationpip install scrapli_replaySee thedocsfor other installation methods/details.A Simple (Pytest) Example@pytest.mark.scrapli_replaydeftest_something_else():withIOSXEDriver(**MY_DEVICE)asconn:result=conn.send_command("show run | i hostname")
Package: scrapper-boilerplate
scrapper-boilerplateScrapping functions and for making easy for further scrappinginstalation(Optional) install enviromentmake sure that you have python installedthen install:pipinstallvirtualenvthen install the enviroment and activate:python-mvenvvenv.\venv\scripts\activateInstall the dependêncies:Install chromedriverclick here, and get the path location.Create .env file with:chromedriver locationtoken of telegram bot (if you use telegram function)Example provided in ".env.example"install the packages:pipinstall-rrequirements.txtand activate:pythonindex.pysetup enviromentcreate ".env" file with following lines:CHROMEDRIVER_PATH="Insert your chromedriver path"
TELEGRAM_TOKEN="Insert your telegram token..."Usagefromtimeimportsleepfromutils.setupimportsetSeleniumimportosROOT_DIR=os.path.dirname(os.path.abspath(__file__))defmain():# import selenium initialization and browser itdriver=setSelenium()driver.get('https://www.google.com')sleep(10)driver.quit()if__name__=="__main__":main()
Package: scrapper-cf
This library contains many useful function which will speed up the making of scrappers. The library focusses on efforts to ensure that the scrappers don't get blocked by the servers. It has functions which:Returns a random user agent out of 10,000 user agents. The user agents are stored in a csv file inside the package.Randomly hit various URLs provided by the usersRaise Alert in the form of e-mail and/or beeps
Package: scrapperDeribit
TODO: make readmeInstallationPython 3.10 or later. To install run:pipinstallscrapperDeribitConfiguration explainedEach run of DeribitClient will take configuration file as input.mysql:host - mySQL hostuser: mySQL user with write accesspassword: mySQL passworddatabase: mySQL db nameuse_bathes_to_record: unused, will be deprecated soonreconnect_max_attempts: number of reconnect try on DB errorsreconnect_wait_time: time in sec before new reconnect tryhdf5:hdf5_database_directory: string with hdf5 storage pathrecord_system:use_batches_to_record: True or False. Unable batch systemnumber_of_tmp_tables: number of circular batch tablessize_of_tmp_batch_table: number of lines in one tableinstrumentNameToIdMapFile: unused, will be deprecated soon \clean_database_at_startup: True or False CleanUp on startuser_data:test_net:client_id: deribit client idclient_secret: deribit client secretproduction:client_id: Noneclient_secret: NoneexternalModules:add_order_manager: True or Falseadd_instrument_manager: True or FalseorderBookScrapper:scrapper_body: right now [OrderBook, Trades, OwnOrderChange, Portfolio] available. You can use only some of themdepth: order book depth/ Highly recommend to use Union[1, 10, 100]test_net: True or Falsecurrency: BTC or ETH or SOLenable_traceback: False # False - defaultenable_database_record: True or False. Enable database record?clean_database: False unused, will be deprecated soonhearth_beat_time: default 60. connection ping-pong time \
group_in_limited_order_book: none (better don't touch)raise_error_at_synthetic: True or False. Should scrapper raise error when underlying for maturity is synthetic?logger_level: WARN | INFO | ERROR. Logger level. INFO can broke buffer when full surface collectingselect_all_order_book: Trueonly_api_orders_processing: Truedatabase_daemon: mysql | hdf5add_extra_instruments: example ['BTC-5MAY23-28000-C', 'BTC-5MAY23-28000-P', 'BTC-PERPETUAL']. List of instruments that should be collecteduse_configuration_to_select_maturities: False used in several scripts with pre-selected configuration about maturities. Will be deprecated soonmaturities_configuration_path: "Maturities_configuration.yaml" Will be deprecated soonDocumentationTODO: add documentation coverage badgeTestingTODO: add testing coverage badgeTODO: link to repo with ZeroMQ test serverExample (scrapping data)TODO: addExample (make and connect own subscription)TODO: addExample (make and connect own strategy)TODO: addPR PipelineTODO: add
Package: scrappie
Scrappy provides a minimal python binding to some of the core functionality of
thescrappie rawbasecaller. The performance of scrappy should be comparable
(if not essentially identical) to scrappie.InstallationScrappy is available on pypi. For linux it is available as a pre-compiled
wheel, for MacOS the distribution will be built from source. Scrappy can be
installed simply as:pip install scrappieNote the spelling! The above may fail on MacOS, due to various compiler and
library versions which might be present. In this case please report an issue
on github.Building from the source is also supported, providing one has met the
requirements for building the scrappie itself. It is recommended to first
ensure scrappie builds cleanly before attempting to build it’s python
counterpart. To build from the source, run:pip install -r requirements
python setup.py installThe second command may be modified as appropriate (perhaps by adding–userfor non-root installations).APIScrappy directly exposes the key parts of scrappie’s “from raw” basecalling
pipeline. The steps are:scrappy.trim_raw: attempt to remove data points corresponding to adapters,scrappy.scale_raw: perform a robust normalization of the data,scrappy.calc_post: calculate time-resolved label probabilties (“posterior probabilities”),scrappy.decode_post: a Viterbi decoding to obtain optimal base sequence.The above is illustrated inscrappy.basecall_raw.Additionally the squiggle simulation and alignment functionality is exposed via:scrappy.sequence_to_squiggle: simulate a squiggle from a base sequence,scrappy.map_signal_to_squiggle: align raw data to a simulated sequence.DemoAfter installation the scrappy program can be run to demonstrate the API’s
function. The program takes one or more filepaths to.fast5files and outputs
FASTA formatted basecalls to stdout, e.g.:scrappy file1.fast5 <file2.fast5> <file3.fast5> …As with scrappie to achieve best performance it is recommended to disable
threading in your BLAS library (this is true also if building an application
using the API).
Package: scrappier
ScrappierScrappier is a web scrapper which uses chrome in a headless mode. This library provides an easy-to-read syntaxis to navigate through the different elements and perform actions.Requirementspython 3.8chrome driver installed in /usr/bin/chromedriverInstalationpip install scrappierBasic usagefromscrappierimportBrowserbrowser=Browser()cards=browser.where_class("card").get()forcardincards:span=card.where_tag_name("span").first()print(span.text())Available methods for browserbuild()html()resize(width:int, height:int)screen(path:str)select(name:str, value:str)url()visit(url:str)wait(seconds:int)webdriver()where_inner_text(text)where_xpath(xpath:str)where_id(id:str)where_name(name:str)where_contain_text(name:str)where_class_name(name:str)where_tag_name(name:str)where_attribute()width()Available methods for ElementFinderuntil(seconds:int)get()first()where_xpath(xpath:str, driver, element=None)where_id(id:str, driver, element=None)where_inner_text(text:str, driver, element=None)where_contain_text(text, driver, element=None)where_class_name(name:str, driver, element=None)where_tag_name(name:str, driver, element=None)next_sibling(name:str, driver, element)Available methods for Elementattributes() -> listenter()type(text:str)text()html()attribute(name:str)click()children()next_sibling()value(value:str = None)gets the value of the element if there is no argument, but if an string was given, will set the input valuewhere_tag_name(name:str)where_attribute(attribute:str, value:str)ColaboratorsIf you are a collaborator, please consider do the next:Create your new functionalityCreate a test of your new functionalitychange the version of the package in setup.pyexecute the next command:python3 setup.py sdist bdist_wheelupload the package:source venv/bin/active && twine upload dist/*<your-version>*for developersinstall twine:python3 -m pip instal twineinstall setuptools:pip3 install setuptoolsinstall setuptools:python3 setup.py sdist bdist_wheelupload your changes:twine upload dist/*
Package: scraprom
scrapromscrapy stats collector for prometheushow to usepip3 install -U scrapromSTATS_CLASS = 'scraprom.PromStatsCollector'SCRAPROM_PUSHGATEWAY_URL = 'host:port'    # default is '0.0.0.0:9091'SCRAPROM_JOB_NAME = 'job_name'    # default is scrapySCRAPROM_PUSH_TIMEOUT = 'timeout_in_seconds'  # default is 3SCRAPROM_UPDATE_INTERVAL = 'interval_in_seconds' # default is 5SCRAPROM_METRICS_PREFIX = 'metrics_prefix'    # default is 'scraprom'
Package: scraproxy
ScraproxyProxy listing library useful for using with pythonrequests.Usage>>> import requests
>>> from scraproxy import Scraproxy

>>> sp = Scraproxy()
>>> sp.fetch_proxies(only_https=True, allow_transparent=False, country='US')

>>> len(sp.proxies)    
>>> requests.get('https://httpbin.org/anything', proxies=sp.random_proxy())
Package: scrapse
ScrapSEPackage descriptionScrapSE downloads and manage the judgments.Currently supported platforms: LEGGI D'ITALIA PA.Install scrapsepip install scrapseThe package creates thescrapsefolder in/Users/your_username, where it will save all judgments in theappropriate subfolders.How to useSaving cookies - important!scrapse leggitalia save-cookie 'your_cokies'This command saves session cookies in a special file, containingyour_cookie.Show filter valuesscrapse leggitalia show-filtersThis command shows the possible values to be assigned to sentence search filters.Download the judgmentsMake sure you havesavedplatform-related cookies before downloading the judgments!.scrapse leggitalia scrap-judgments -l torino -s 'Sez. lavoro, Sez. V'This command creates a folder in/Users/your_username/scrapse/leggitalianamedsez.lavoro&sez.v_torinocontaining the judgments.Dump judgments to json formatscrapse leggitalia dump-judgments -d 'folder_path'This command creates the json files by saving them in the/Users/your_username/scrapse/leggitalia/judgments_dumpfolder.For more helpFor more information for each command.command-name --help