Saravanan666/pypi-org huggingface dataset:
Package: webrunner
WebRunner is a pythonic module used for web scrapping and testing.Here are simple instructions about how to use webrunner.First of all, import WebBrowser to your namespace and instanciate it.>>> from webrunner import WebBrowser
>>> wb = WebBrowser()Now, you can use the method urlopen for open some url on the web>>> wb.urlopen('www.google.com')Now, we can ‘see’ the google’s page
>>> g_page = wb.current_pageLet’s do a search
>>> form = g_page.forms[‘f’]
>>> form.set_value(‘some search’, ‘q’)
>>> wb.submit_form(form)
>>> results_page = wb.current_page
Package: web-rwkv-py
Web-RWKV-PyPython binding forweb-rwkv.TodosBasic V5 inference supportSupport V4, V5 and V6Batched inferenceUsageInstall python and rust.Install maturin by$pipinstallmaturinBuild and install:$maturindevelopTry usingweb-rwkvin python:importweb_rwkv_pyaswrpmodel=wrp.v5.Model("/path/to/model.st",# model pathquant=0,# int8 quantization layersquant_nf4=0,# nf4 quantization layersturbo=True,# faster when reading long promptstoken_chunk_size=256# maximum tokens in an inference chunk (can be 32, 64, 256, 1024, etc.))logits,state=wrp.v5.run_one(model,[114,514],state=None)Advanced UsageMove state to host memory:logits,state=wrp.v5.run_one(model,[114,514],state=None)# returned state is on vramstate_cpu=state.back()Load state from host memory:state=wrp.v5.ModelState(model,1)state.load(state_cpu)logits,state=wrp.v5.run_one(model,[114,514],state=state_cpu)Return predictions of all tokens (not only the last's):logits,state=wrp.v5.run_one_full(model,[114,514],state=None)len(logits)# 2
Package: web-sand
SANDTable of ContentsIntroductionInstallationIntroductionSAND is an application to annotate semantic descriptions of tables and (optionally) linked records in tables to a target knowledge graph, then it can automatically export the table data to RDF, JSON-LD, etc. It also does basic data cleaning automatically based on the annotated semantic descriptions. SAND is designed to be customizable: you can plug in a new semantic modeling algorithm (which generates a semantic description automatically) or different knowledge graphs as long as you have a suitable plugin implemented SAND's plugin interface.Moreover, SAND offers an internal KG browsing and table filtering so you can interactively browsing and modeling your tables.For a demo, please see: ourdemo paper,demo video.InstallationInstall from pip:pip install -U web-sandUsageInitialize database:sand init -d <dbfile>. For example:sand init -d ./data/sand.dbStart the webserver:sand start -d <dbfile> --externaldb <folder_of_ent_and_ont_db>Open the URL:http://localhost:5524For example, checkoutserver.shDevelopmentcd towwwInstallyarnandyalcInstall dependencies:yarn installStart development server:yarn start. Then, access development server at:http://127.0.0.1:3000.Build production files:yarn buildBuild library files and publish to private index (only if you are releasedsandas a library):yarn build:lib && yalc public --private
Package: websauna.blog
This is a Python package for websauna.blog, an addon forWebsauna framework. You can use it as a simple blog for your site or as an example of Websaunsa design patterns.FeaturesScreenshotsBlog postAdminInstallationAdding to your siteLocal development modeRunning the development websiteRunning the test suiteMore informationChangelog for Websauna Blog1.0a2 (2018-04-22)1.0a1 (2018-03-06)FeaturesMarkdown based editingDisqusbased commentingPost description and meta for Google (SEO), Facebook, Twitter. Sitemap support and Google Article metadata support.Blog post management through Websauna admin interfaceDrafts (admin only visible) and published postsRSS feedBasic unit and functional test suiteNote that this addon is not intended to be used as is, but more of an example. You most likely want to fork it over and modify for your own needs.ScreenshotsBlog postAdminInstallationAdding to your siteInclude addon in your application initializer:classInitializer(websauna.system.Initializer):definclude_addons(self):"""Include this addon in the configuration."""self.config.include("websauna.blog")Example settings:# Title on blog rollblog.title=My little Websauna blog# this is "websauna" part from websaua.disqus.com/embed.js univeral# embed linkblog.disqus_id=websauna# The email appearing in RSS feed# (It is recommended not to use any real email)blog.rss_feed_email=no-reply@example.comSeenav.htmlexample how to add a link to the blog in your site navigation.Add RSS feed discovery by customizingsite/meta.htmltemplate:{# Misc ``<meta>`` tags in ``<head>``. #}<metahttp-equiv="X-UA-Compatible"content="IE=edge,chrome=1"><metaname="viewport"content="width=device-width"/>{% include "blog/rss_head.html" %}Create migrations for blog post SQL table for your site:ws-alembic -c myapp/conf/development.ini -x packages=all revision --auto -m "Adding blog content types"Run migrations:ws-alembic -c myapp/conf/development.ini -x packages=all upgrade headGo to admin, start adding blog posts.Local development modeActivate the virtual environment of your Websauna application.Then:cd blog  # This is the folder with setup.py file
pip install -e .Running the development websiteLocal development machineExample (OSX / Homebrew):psql create blog_dev
ws-sync-db websauna/blog/conf/development.ini
ws-create-user websauna/blog/conf/development.ini admin@example.com mypassword
ws-pserve websauna/blog/conf/development.ini --reloadRunning the test suiteFirst create test database:# Create database used for unit testing
psql create blog_testInstall test and dev dependencies (run in the folder withsetup.py):pip install -e ".[dev,test]"Run test suite using py.test running:py.testMore informationPlease seehttps://websauna.org/Changelog for Websauna Blog1.0a2 (2018-04-22)ClosesIssue #6: missing ‘namespace_packages’, .1.0a1 (2018-03-06)Initial Release
Package: websauna.j2secret
This is a Jinja2 extension required by Cookiecutter Websauna templates.InstallationUsageRunning the test suiteMore informationChangelogInstallationUsing the same virtualenv where cookiecutter is installed install this extension executing pip install:pipinstallwebsauna.j2secretUsageExtend the Cookiecutter environment with this customJinja2 extensions,
that adds a tag namedsecret, specifing thewebsauna.j2secretextension incookiecutter.jsonas follows:{"authentication_random":"{% secret() %}","authomatic_random":"{% secret() %}","session_random":"{% secret() %}","_extensions":["websauna.j2secret.SecretExtension"]}Read more aboutTemplate Extensions for Cookiecutter.Running the test suiteInstall test and dev dependencies (run in the folder withsetup.py):pip install -e ".[test]"Run test suite using py.test running:py.testMore informationPlease seehttps://websauna.org/Changelog1.0.0a1 (2017-10-16)Implement Jinja2 tagsecretto be used with Cookiecutter templates.
Package: websauna.magiclogin
This is a Python package for magiclogin, an addon forWebsauna framework.To run this package you need Python 3.5.2+, PostgresSQL and Redis.FeaturesPasswordless login with OAuth and email link options.Throttle protection against brute forcing and spamInterstitial page to require user to login and then continue what ever HTTP GET/POST action user was performingScreenshotsInstallationSetup OAuth credential indevelopment.inianddevelopment-secrets.iniaccording to Websauna documentation.Exampledevelopment.ini:websauna.social_logins =
    facebook
    googleExampledevelopment-secrets.inibits:[facebook]
class = authomatic.providers.oauth2.Facebook
consumer_key = xxx
consumer_secret = yyy
scope = user_about_me, email
mapper = websauna.system.user.social.FacebookMapper

[google]
class = websauna.system.user.googleoauth.Google
consumer_key = xxx
consumer_secret = yyy
mapper = websauna.system.user.social.GoogleMapper
scope = profile emailCopy bits fromdemo.pyto your application initializer.SettingsAvailable INI settings:# Throttle email login endpoints to this window
magiclogin.login_email_throttle = 50/3600

# How fast email login link dies
magiclogin.email_token_expiration_seconds = 300

# Text shown on the login panel
magiclogin.login_slogan = Your login text goes hereYou might also have long, secure, sessions in production:# Set session length to one year
redis.sessions.cookie_max_age = 31536000
redis.sessions.cookie_secure = True
redis.sessions.cookie_httponly = TrueRunning the development websiteLocal development machineExample (OSX / Homebrew):createdb magiclogin_dev
ws-sync-db  ws://websauna/magiclogin/conf/development.ini
pserve ws://websauna/magiclogin/conf/development.ini --reloadYou can visit:http://localhost:6543/login

http://localhost:6543/require_login_example_pageRunning the test suiteFirst create test database:# Create database used for unit testing
psql create magiclogin_testInstall test and dev dependencies (run in the folder withsetup.py):pip install -e ".[dev,test]"Run test suite using py.test running:py.testMore informationPlease seehttps://websauna.org/Changelog for Websauna Magic Login1.0a2 (2018-04-22)Fix missing ‘namespace_packages’.1.0a1 (2018-03-06)Initial Release
Package: webscaff
https://github.com/idlesign/webscaffDescriptionRemote scaffolding and orchestration for web applicationsUsed stackDebian-based OS (Ubuntu 18.04, 20.04, 22.04 tested) as a basis.Gitfor source code updates.Systemdto securely run your webservice.PostgreSQLas a reliable DBMS.uWSGIas a platform (handling routing, static, background tasks, etc.).Python 3to cover your needs.Djangoas a rich and solid webframework.And also:Certbotintegration for free HTTPS certificates (webroot plugin).SSH Agent forwarding for project code updates on remote using keys from your system.A taste of itAfter install thewebscaffcommand is available.;Werolloutprojectskeletonusing`makeapp`.$makeappnewmyproject-d"My webscaff project"-twebscaff/home/some/here;Switchintoprojectdirectorycontaining`wscaff.yml`whichisusedbywebscaff.$cd/home/some/here;Getbasicinformationabouttheremote.$webscaffinfo;Preparetheremoteforyouwebapplication.$webscaffrun.initializeWebscaff offers a number of useful command, among them:;Rolloutanewversionofyorprojectfromrepository.$webscaffrollout;Makeabackupanddownloadtolocaldirectory.$webscaffrun.backupMore commands are available.Detailed instructions could be found in the documentation.Documentationhttp://webscaff.readthedocs.org/
Package: webschaber
ï»¿# WebSchaberThe WebSchaber is Python3 script which extracts the text and images content on search engine `bing.com`It helps the user in a way that the user will be getting only meaningful results and images for their search query. It does not download the ad content and hence saving data for the user.It requests for a search term and creates directory (if not made previously) in the root directory of the script where all the content of the related particular search is stored. This python script will be downloading the hypertext and hyperlink to that text and saving it to a .txt file within the directory made by the script itself. This directory saves the text content as well as the images downloaded using the script.## Requirements`1.	Modules:a. requests: Used for requesting various content through two possible HTTPS Methods: GET and POST.This project uses GET Method.b. BeautifulSoup: Used for creating JSON like dictionary using HTML Parser. This project uses bs4.c. os: Used for checking and making directories.d. PIL.Image: Used for extracting image content.e. io.ByteIO: Used for saving the extracted image using the PIL.Image.2.	Internet Connection: Continuous high speed internet connection is required for the proper function of the python script as it           continuously creates the copy of the images to the local machine.`## Installation`pip install webschaber`
Package: webscp
# Python Image Scraper## IntroductionPython Image scrapper can help in gathering huge dataset of images for any custom keyword. This helps save time and gather unique images for any keyword. Gathered images will be uploaded to [anonfiles.com](https://anonfiles.com) and the link will be emailed to the user.## Demo
![](static/images/demo.gif)## FeaturesScraping Data in Multiple ThreadsAuto Execute Project AnytimeUploads files to [anonfiles.com](https://anonfiles.com)Emails link after scraping imageslogs every action performed by scrapper## Installation### Using Docker
1.  Clone the project using`sh git clonehttps://github.com/govind2220000/image_scraper.git`2.  Renameexample.config.initoconfig.ini. Fill it with appropriate values.3.  Build the project using Docker`sh cd image_scraper sudodocker-composebuild `4.  Deploy the project`sh sudodocker-composeup `### Using Python
1. Clone the project using`sh git clonehttps://github.com/govind2220000/image_scraper.git`2. Install dependencies`sh cd image_scraper pip install-rrequirements.txt `3. Run application`sh python3-msrc `## Documentation Links
High Level Design Document : [View / Download](https://drive.google.com/file/d/1xWUd1qql4b25_gpqhOVHEoX8_NvHJaic/view?usp=sharing)Low Level Design Document  : [View / Download](https://drive.google.com/file/d/1BO-RErAp7n9-cYCCm36wpSba0sOug4pd/view?usp=sharing)Project Report             : [View / Download](https://drive.google.com/file/d/1YDMSYhX_ldxOHqkgJtY31nyVLZivBVWL/view?usp=sharing)WireFrame Document         : [View / Download](https://drive.google.com/file/d/1IIiREJi5Jaa4wPbiQ2gAZkvDBF1q5PQ2/view)## Authors
- [Govind Choudhary](https://github.com/govind2220000)
- [Sreejith Subhash](https://github.com/sjith7)
Package: web-scraper
This package include modules for findng links in a webpage and its children.In the main modulefind_links_by_extensionlinks are found using two sub-modules and then added together:Using Google Search Results (get_links_using_Google_search)Since we can specify which types of files we are looking for when we search in Google, this methos scrapes these results.
But this method is not complete:Google search works based on crawlers, and sometimes they don’t index properly. For example [this][1] webpage has three pdf files at the moment (Aug 7 2018), but when we [use google search][2] to find them it finds only two  although the files were uploaded 4 years ago.It doesn’t work with some websites. For example [this][3] webpage  has three pdf files but google [cannot find any][4].If many requests are sent in a short period of time, Google blocks access and asks for CAPTCHA solving.Using a direct method of finding all urls in the given page and following those links if they are refering to children pages and seach recursively (get_links_directly)While this method does not miss any files in pages that it gets to (in contrast to method 1 which sometimes do), it may not find all the files because:Some webpages in the domain may be isolated i.e. there is no link to them in the parent pages. For these cases method 1 above works.In rare cases the link to a file of type xyz may not have .xyz in the link ([example][5]). In these cases method 2 cannot detect the file (because it only relies on the extesion appearing in the links), but method 1 detects correctly in these cases.So the two methods complete each other’s gaps.[1]:http://www.midi.gouv.qc.ca/publications/en/planification/[2]:https://www.google.com/search?q=site%3Ahttp%3A%2F%2Fwww.midi.gouv.qc.ca%2Fpublications%2Fen%2Fplanification%2F+filetype%3Apdf[3]:http://www.sfu.ca/~vvaezian/Summary/[4]:https://www.google.com/search?q=site%3Ahttp%3A%2F%2Fwww.sfu.ca%2F~vvaezian%2FSummary%2F+filetype%3Apdf[5]:http://www.sfu.ca/~robson/Random
Package: web-scraper-python-library
Web Scraping - Python LibraryA Python library that lets you easily scrape data from popular websites using basic product informationOverviewFor many, collecting product data can be helpful for monitoring price changes or helping decide which e-commerce site to purchase from. However, creating a web scraper from scratch can be cumbersome and time consuming. My goal is to make it easier for people to collect product data, and this Python library aims to simplify the web scraping process. With basic inputs like product information and store url, you can have easy access to rich product information.InstallationTo install, run the following:pip install web-scraper-python-libraryUsageProduct SearchThe following code will retrieve and print the product data for aniphone 12fromAmazonas a JSON object.product: a product name, like you would put into the product search page of a company's websitecompany: 'eBay', 'Walmart', or 'Amazon'Codefromweb_scraper_python_libraryimportmainasmproduct_data=m.main(product='iphone 12',company='Amazon')print(product_data)Output[{"name":"Apple iPhone 12, 64GB, Green - Fully Unlocked (Renewed)","price":390.0,"extraction_date":"2023-03-23 17:38:32"},...{"name":"Apple iPhone 12 Mini, 64GB, Black - Unlocked (Renewed)","price":312.0,"extraction_date":"2023-03-23 17:38:32"}]
Package: webscraperr
WebscraperrThis Python library is designed to facilitate the common workflow of web scraping, particularly for e-commerce websites. It provides a structured framework where users can define their own logic for gathering product URLs, parsing individual product pages, and selecting the next page. The URLs and product info are saved directly to a database. It supports various databases such as SQLite and MySQL.InstallationInstall webscraperr with pippipinstallwebscraperrUsageThe configurations of the scraper is stored in a config dictionary. The config must be prepared, modified and validated before passing it to the scraper.fromwebscraperr.configimportget_default_config,validate_config,DBTypesconfig=get_default_config()config['DATABASE']['TYPE']=DBTypes.SQLITEconfig['DATABASE']['DATABASE']='mydatabase.db'config['DATABASE']['TABLE']='products'# If TABLE is not set "items" will be the defaut table nameconfig['SCRAPER']['REQUEST_DELAY']=1.6validate_config(config)# Will raise an error if config is not properly setAfter preparing and validating the config, you must initialize the databasefromwebscraperr.dbimportinit_sqliteinit_sqlite(config['DATABASE'])# This will create the database and the tableFor this example we are going to useWebScraperRequest. This scrapper will be usingrequestslibrary for the http requests. You will need to define the functions for parsing the html. There is alsoWebScraperChromethat usesselenium-wireandundetected-chromedriver.fromwebscraperrimportWebScraperRequestfromurllib.parseimporturljoinimportparselurls=["https://webscraper.io/test-sites/e-commerce/static/computers/tablets"]# The `get_next_page_func` must return a url or None. If it returns None it means there is no next pagedefget_next_page_func(response):selector=parsel.Selector(text=response.text)# in this example `parsel` is used for parsing the htmlnext_page_url=selector.css('a[rel="next"]::attr(href)').get()ifnext_page_urlisnotNone:returnurljoin(BASE_URL,next_page_url)returnNone# The `parse_info_func` must return a `dict`.defparse_info_func(response):selector=parsel.Selector(text=response.text)info={'name':selector.css(".caption h4:nth-child(2)::text").get(),'price':selector.css(".caption .price::text").get()}returninfowithWebScraperRequest(config)asscraper:scraper.get_items_urls_func=lambdaselector:[urljoin(BASE_URL,i)foriinselector.css(".thumbnail a::attr(href)").getall()]scraper.get_next_page_func=get_next_page_funcscraper.parse_info_func=parse_info_funcscraper.scrape_items_urls(urls)# This will start the scraping of products urlsscraper.scrape_items_infos()# This will navigate to the product page and parse the htmlDevelopment StatusPlease note that this library is still under development and may be subject to changes. I am constantly working on improving its functionality, flexibility and performance. Your patience, feedback, and contributions are much appreciated.
Package: webscrapetools
This package lets you efficiently retrieve pages from the Internet by caching request’s results.Basic commandsImporting required modules first:from webscrapetools import urlcachingInitializing the cache:urlcaching.set_cache_path(‘.wst_cache’)The option _expiry_days_ sets the cache expiry period, default is 10 days.This is a required step: otherwise responses to url calls will simply not be cached.
Cache data are stored in the specified folder, so that re-using the same string makes the cache persistent. This creates
the folder on the fly if it does not exist.
The following command cleans up the cache, making sure we start with no prior data:urlcaching.empty_cache()Opening an url with the following command stores the repsonse content behind the scene, so that subsequent calls will
not hit the network.urlcaching.open_url(‘http://www.google.com’)Full examplefromwebscrapetoolsimporturlcachingimporttime# Initializing the cacheurlcaching.set_cache_path('.wst_cache')# Making sure we start from scratchurlcaching.empty_cache()# Demo with 5 identical calls... only the first one is delayed, all others are hitting the cachecount_calls=1whilecount_calls<=5:start_time=time.time()urlcaching.open_url('http://deelay.me/5000/http://www.google.com')duration=time.time()-start_timeprint('duration for call{}:{:0.2f}'.format(count_calls,duration))count_calls+=1# Cleaning upurlcaching.empty_cache()The code above outputs the following:duration for call 1: 6.74
duration for call 2: 0.00
duration for call 3: 0.00
duration for call 4: 0.00
duration for call 5: 0.00Example plugging in a custom clientThe framework lets you customize the way you access the web. It is therefore possible to drive a browser
via Selenium for example.fromwebscrapetoolsimporturlcachingurlcaching.set_cache_path('./output/tests',max_node_files=10,rebalancing_limit=100)defdummy_client():returnNonedefdummy_call(_,key):return'{:d}'.format(int(key))*int(key),key# simulating calls using the dummy clientkeys=('{:05d}'.format(count)forcountinrange(500))forkeyinkeys:urlcaching.open_url(key,init_client_func=dummy_client,call_client_func=dummy_call)urlcaching.empty_cache()
Package: webscrapex
A comprehensive web scraping pipeline for extracting and storing real estate dataPurpose of the packageThe primary objective of this package is to provide an efficient solution for web scraping tasks. It has essentially functionalities including link extraction, data extraction, data cleaning, and data storage to database.Features- Date                      - Bathrooms  
- Build Year                - Car Parking
- Floors                    - Ancillary
- Sitting Rooms             - LandSize
- Dining Rooms              - Price
- Bedrooms                  - District
- Wardrobes                 - SectorInstallationTo install the package, run the following command:!pipinstallWebScrapeXContributionContributions are welcome. If you encounter any bugs or have suggestions for improvements, please let me know atinyangel@yahoo.com. ThanksAuthorThis package was developed by Lisa Yvette INYANGE (https://github.com/ILisa250)LicenseThe package is released under the MIT license. (https://choosealicense.com/licenses/mit/)DependenciesThe package has the following dependencies:Python Decouple: Used for managing settings and configuration.Python Dotenv: Used for loading environment variables from a .env fileScraping URLsThe package supports scraping the following types of real estate listings from the Imali.biz website:Apartment for Sale: https://imali.biz/category/1/125/search?pg=
Apartment for Rent: https://imali.biz/category/0/91/search?pg=
House for Rent: https://imali.biz/category/0/27/search?pg=
House for Sale: https://imali.biz/category/0/24/search?pg=Usage exampleHere's an example of how to use the WebScrapeX package to scrape, clean, and save real estate data:fromWebScrapeXimportscrape_clean_save_data
importosenv_path=os.path.abspath('.env')# Specify the link of the real estate type to scrapeurl="https://imali.biz/category/1/125/search?pg="# Specify the name of the file to save the data (in lowercase)file_name="real_estate_data.csv"# Scrape, clean, and save the datascrape_clean_save_data(url,file_name,env_path)Note: File name should be either "house_sale" or "house_for_rent" or "apartment_for_sale" or "apartment_for_rent".