Saravanan666/pypi-org huggingface dataset:
Package: os-scrapy-ipfilter-resolver
os-scrapy-ipfilter-resolverThis project provide a DNS Resolver for ip blacklist/whitelist and config DNS expire time.Installpip install os-scrapy-ipfilter-resolverYou can run example spider directly in the project root path.scrapy crawl exampleUsageSettingsenable DNS Resolver, in the project settings.py file:DNS_RESOLVER = "os_scrapy_ipfilter_resolver.Resolver"config IP blacklist, it will raiseIPFilteredExceptionwhen the request ip in the blacklistIP_DISALLOWED = ["192.168.0.0/16", "10.143.0.1"]config IP whitelist, priority greater than blacklistIP_ALLOWED = ["192.168.0.1"]config DNS cache size and expire time(seconds)DNSCACHE_ENABLED = True
DNSCACHE_SIZE = 10000
DNSCACHE_EXPIRE = 24 * 60 * 60config DNS lookup timeout(seconds)DNS_TIMEOUT = 60Unit TeststoxLicenseMIT licensed.
Package: os-scrapy-kafka-pipeline
os-scrapy-kafka-pipelineThis project provide pipeline to send Scrapy Item to kafka as JSON formatFeatures:support config default kafka brokers and topic in the settings.py filesupportkafka-pythonproducerinit argssupport dynamic connect and send to other kafka cluster and topic using item metaitem will send to kafka as JSON format, bytes can be encoded to base64 string if it can not be utf-8 encodedInstallpip install os-scrapy-kafka-pipelineYou can run example spider directly in the project root path.scrapy crawl exampleUsageSettingsEnable pipeline in the project settings.py fileITEM_PIPELINES = {
    "os_scrapy_kafka_pipeline.KafkaPipeline": 300,
}Config default kafka brokersKAFKA_PRODUCER_BROKERS = ["broker01.kafka:9092", "broker02.kafka:9092"]brokers in the item meta will override this default valuepipeline will not be enabled when this settings can not to start kafka connectionit will raise exception when no brokers configuredConfig default kafkaproducerKAFKA_PRODUCER_CONFIGS = {"client_id": "id01", "retries": 1}this is global config, the dynamic connections will use this configsthebootstrap_serverswill not work whenKAFKA_PRRDUCER_BROKERSalready configuredConfig defult topicKAFKA_PRODUCER_TOPIC = "topic01"the config in the item.meta will override this configit will raise exception when no topic configuredConfig kafka-python loglevel (default "WARNING")KAFKA_PRODUCER_LOGLEVEL = "DEBUG"Config kafka producer close timeout (default: None)KAFKA_PRODUCER_CLOSE_TIMEOUT = 10Ensure base64The bytes type of the item mumber will be decoded by utf-8, if decode fail, the pipeline will use base64 to encode the bytes when you set:KAFKA_VALUE_ENSURE_BASE64 = TrueFilter fieldYou can filter item fields which will not export and send to kafkaKAFKA_EXPORT_FILTER = ["filtered_field"]Dynamic Kafka Connection with item.metayou can set topic, key, partition using item["meta"]the item must has meta mumber which type is dictoptions:meta = {
    "kafka.topic": "topic01",
    "kafka.key": "key01",
    "kafka.partition": 1,
    "kafka.brokers": "broker01.kafka:9092,broker02.kafka:9092"
}Storage FormatItem will send to kafka as JSON format, bytes will encode to base64Unit Testssh scripts/test.shLicenseMIT licensed.
Package: os_scrapy_linkextractor
# os_scrapy_linkextractorScrapy project auto generated by [os-scrapy-cookiecutter](https://github.com/cfhamlet/os-scrapy-cookiecutter).## ProjectThis scrapy project is a python package generated by os-scrapy-cookiecutter.You can run it as normal scrapy project with scrapy commands.It is also a python package, the scaffolds can be used for formatting, testing, installing.lint` sh scripts/lint.sh `test` sh scripts/test.sh `install` python setup.py install `exampleYou can run example spider:` scrapy crawl example `files` . ├── os_scrapy_linkextractor # scrapy project │   ├── __init__.py │   ├── items.py │   ├── middlewares.py │   ├── pipelines.py │   ├── settings.py │   └── spiders │       └── __init__.py ├── LICENSE # license file ├── MANIFEST.in # setup install package file ├── pytest.ini # pytest config file ├── README.md ├── requirements │   ├──requirements-lint.txt# requirements of format tools used by scripts/lint.sh and tox │   ├──requirements-test.txt# requirements of test used by tox │   └── requirements.txt # requirements of this project ├── scrapy.cfg # scrapy project entrypoint file ├── scripts │   ├── test.sh # run test │   └── lint.sh # format the source file ├── setup.cfg # setup tool config file ├── setup.py # setup tool ├── tests │   ├── __init__.py │   └── test_scrapy_check.py # unit test with scrapy check └── tox.ini # tox config file `## Install` python setup.py install `## Usage` scrapy-h`## Unit Teststox## LicenseMIT licensed.
Package: os-scrapy-random-proxy
os_scrapy_random_proxyScrapy project auto generated byos-scrapy-cookiecutter.ProjectThis scrapy project is a python package generated by os-scrapy-cookiecutter.You can run it as normal scrapy project with scrapy commands.It is also a python package, the scaffolds can be used for formatting, testing, installing.lintsh scripts/lint.shtestsh scripts/test.shinstallpython setup.py installexampleYou can run example spider:scrapy crawl examplefiles.
├── os_scrapy_random_proxy # scrapy project          
│   ├── __init__.py
│   ├── items.py
│   ├── middlewares.py
│   ├── pipelines.py
│   ├── settings.py
│   └── spiders
│       └── __init__.py
├── LICENSE # license file
├── MANIFEST.in # setup install package file
├── pytest.ini # pytest config file
├── README.md
├── requirements
│   ├── requirements-lint.txt # requirements of format tools used by scripts/lint.sh and tox
│   ├── requirements-test.txt # requirements of test used by tox
│   └── requirements.txt # requirements of this project
├── scrapy.cfg # scrapy project entrypoint file
├── scripts
│   ├── test.sh # run test 
│   └── lint.sh # format the source file
├── setup.cfg # setup tool config file
├── setup.py # setup tool 
├── tests
│   ├── __init__.py
│   └── test_scrapy_check.py # unit test with scrapy check
└── tox.ini # tox config fileInstallpython setup.py installUsagescrapy -hUnit TeststoxLicenseMIT licensed.
Package: os-scrapy-random-useragent
os-scrapy-random-useragentThis project provide aDownloader Middlewareto change 'User-Agent' of request headers.Installpip install os-scrapy-random-useragentUsageSettingsenable downloader middleware in settings.py file:this middleware will override default Scrapy User-AgentDOWNLOADER_MIDDLEWARES = {
    "os_scrapy_random_useragent.RandomUserAgentMiddleware": 543,
}config useragents:by file:USER_AGENTS = "./your-useragets-file"by string:USER_AGENTS = "Your-Useragent-String"by list:USER_AGENTS = ["User-Agent-01", "User-Agent-02"]Unit TeststoxLicenseMIT licensed.
Package: os-scrapy-record
os-scrapy-recordThis project provide extensions to process Response/Failure, generate standard Item.Installpip install os-scrapy-recordYou can run example spider directly in the project root pathscrapy crawl exampleAPIsos_scrapy_record.ResponseCallbackthecallbackmethod of this extension will replace the defaultRequest.callback, process Response and generate FetchRecordthecallbackmethod will not work when the request already set callback functionthecallbackmethod will override theparsemethod of spiderenable extension in the project settings.py file:EXTENSIONS = {
    "os_scrapy_record.ResponseCallback": 1,
}os_scrapy_record.ResponseErrbacktheerrbackmethod of this extension will replace the defaultRequest.errback, process Failure and generate FetchRecordtheerrbackmethod will not work when the request already set errback functionenable extension in the project settings.py file:EXTENSIONS = {
    "os_scrapy_record.ResponseErrback": 1,
}os_scrapy_record.FetchRecordThis class is subclass ofItemthe mumbers of this class are:request:os_scrapy_record.items.RequestItem, members: url, method, headers, bodymeta:dict, request.meta, it is better to use lower case and '_' as separator as keyresponse:os_scrapy_record.items.ResponseItem，members: headers, body, status, ip_address(Scrapy 2.1.0+), failureos_scrapy_record.fetch_status.FetchStatusA mumber of ResponseItem, include HTTP, DNS, Network and user defined status. It is a two-tuple object: group and code. e.g, HTTP:200, DNS:-2, SERVER:111, RULE:16Unit Testssh scripts/test.shLicenseMIT licensed.
Package: os-scrapy-uvicorn
os-scrapy-uvicornThis project provide a extension to start a ASGI http server(Uvicorn) along with Scrapy in the same process.You can use theASGI framework(recommendFastAPI) to create app to communicate with Scrapy.Require: Python 3.6+, Scrapy 2.0+Installpip install os-scrapy-uvicornYou can run example spider directly in the project root pathscrapy crawl exampleSettingsuse asyncio reactorTWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"or you can useos-scrapy(installed with this project) to start crawling with-rcommand line optionos-scrapy crawl -r asyncio exampleenable extensionEXTENSIONS = {
    "os_scrapy_uvicorn.Uvicron": 1,
}app path, you can use ASGI app frameworks to create your app,FastAPIis recommendedUVICORN_APP = "app_module:app"uvicorn server settings,supported settingsUVICORN_CONFIG = {"host": "0.0.0.0", "port": 5000}when the server started, Scrapy crawler instance is attached to the app, it is the entrypoint to commnicate with ScrapyUnit Testssh scripts/test.shLicenseMIT licensed.
Package: ossd
OpenSubtitles Subtitles Downloader - OSSDThis package allows you to download subtitles as text fromwww.opensubtitles.com. All you need to have isSeleniumwith Chrome driver,requestspackage (pip install requests) and thisOSSDpackage.Instalationpip install ossdPyPi page -https://pypi.org/project/ossd/Usageimport ossd
ossd.get_subtitles(driver, movie_name)driver --Chrome Driver (Selenium)movie_name -- Name of movie which subtitles you want to downloadMethodsget_subtitles-- returns most downloaded subtitles text, if error occurs then returns " " (whitespace) and prints error codeparse_subtitles-- parses script lines from default format into one line only text format, skips ads on the script startExampleimport ossd
 from selenium import webdriver  
 driver = webdriver.Chrome(executable_path=driverPath)
 text = ossd.get_subtitles(driver, "Star Wars: A New Hope")Output (in variable text)Same ashere.000:02:40,680 --> 00:02:42,557Did you hear that? \1 \
00:02:42,680 --> 00:02:45,319They shut down the main reactor. \
We'll be destroyed for sure. \200:02:45,440 --> 00:02:46,873This is madness. \300:02:56,560 --> 00:02:57,913We're doomed.....one_text = ossd.parse_subtitles(text)Output (in variable one_text)Did you hear that? They shut down the main reactor. We'll be destroyed for sure. This is madness. We're doomed. ...Licence - MITFree for use, editing, selling, just mention authorDevelopmentClean ununsed:rm -rf build/ dist/ *egg* **.pyc __pycache__Build package:python setup.py sdist bdist_wheeldeploy package:python -m twine upload dist/*
Package: os-service-types
Python library for consuming OpenStack sevice-types-authority dataTheOpenStack Service Types Authoritycontains information about official
OpenStack services and their historicalservice-typealiases.The data is in JSON and the latest data should always be used. This simple
library exists to allow for easy consumption of the data, along with a built-in
version of the data to use in case network access is for some reason not
possible and local caching of the fetched data.Free software: Apache licenseDocumentation:https://docs.openstack.org/os-service-types/latest/Source:https://opendev.org/openstack/os-service-typesBugs:https://storyboard.openstack.org/#!/project/904Release notes:https://docs.openstack.org/releasenotes/os-service-types/
Package: ossFileField
Django OSS File Fielddjango-oss-file-fieldprovides a Django Direct File Transfer From Server.FeaturesDjango Direct File Transfer From Server to OSSFlexible File Size/Type LimitationProgress Bar DisplayWorks in Python 3InstallationInstall$pipinstallossFileFieldAdd'ossFileField'to yourINSTALLED_APPSsettingAuthentication settingsUse the following settings to authenticate with AliCloud OSS.# AliCloud Access Key IDOSS_ACCESS_KEY_ID=<YourAccessKeyID># AliCloud Access Key SecretOSS_ACCESS_KEY_SECRET=<YourAccessKeySecret># OSS Bucket NameOSS_BUCKET_NAME=<YourOSSBucketname># AliCloud Bucket Host format: http://bucket_name.endpoint eg: http://test.oss-cn-hangzhou.aliyuncs.comOSS_BUCKET_HOST=<YourOssBucketHost># AliCloud Token Expire TimeOSS_EXPIRE_TIME=<TokenExpireTime># Get OSS Token RoutingOSS_TOKEN_ROUTE=<GetOSSTokenRoute># Oss Bucket Proxy URLOSS_PROXY_URL=<OSSBucketProxyURL># Server URLSERVER_URL=<YourAppServerDomainURL># Upload To Oss Dir(end of slash)UPLOAD_DIR=<UploadOSSDir>Get OSS TokenRegister the route in the url file to get the token# urls.pyfromdjango.conf.urlsimporturlfrom.importviewsurlpatterns=[url(r'^get_oss_token$',views.get_oss_token,name='get_oss_token'),]# views.pyfromdjango.httpimportHttpResponsefromdjango.contrib.auth.decoratorsimportlogin_requiredfromossFileField.utilsimportOssToken@login_required@require_http_methods(["GET"])defget_oss_token(request):oss_token=OssToken.get_oss_key()returnHttpResponse(oss_token)Usagemodels.py import OssFileField# models.pyfromossFileFieldimportOssFileFieldclassImages(models.Model):name=models.CharField(max_length=64,blank=False)oss_file=OssFileField(prefix='test/',file_type='image',max_file_size='20m',blank=False)Version1.0.3Support models prefix attribute((end of slash))Support limit uploaded file sizeThe uploaded file name includes the file origin name and file size1.0.4Support public bucketSupport file name with spaces1.0.5Support chinese file name
Package: ossfs
OSSFSis a Python-based interface for file systems that enables interaction with
OSS (Object Storage Service). ThroughOSSFS, users can utilize fsspec’s standard
API to operate on OSS objectsInstallationYou can installOSSFSviapipfromPyPI:$pipinstallossfsUp-to-date package also provided through conda-forge distribution:$condainstall-cconda-forgeossfsQuick StartHere is a simple example of locating and reading an object in OSS.importossfsfs=ossfs.OSSFileSystem(endpoint='http://oss-cn-hangzhou.aliyuncs.com')fs.ls('/dvc-test-anonymous/LICENSE')[{'name':'/dvc-test-anonymous/LICENSE','Key':'/dvc-test-anonymous/LICENSE','type':'file','size':11357,'Size':11357,'StorageClass':'OBJECT','LastModified':1622761222}]withfs.open('/dvc-test-anonymous/LICENSE')asf:...print(f.readline())b'                                 Apache License\n'For more use case and apis please refer to the documentation offsspecAsync OSSFSAsyncOSSFSis a variant of ossfs that utilizes the third-party async OSS
backendaiooss2, rather than the official sync one,oss2. Async OSSFS
allows for concurrent calls within bulk operations, such ascat,put, andgetetc even from normal code, and enables the direct use of fsspec in async
code without blocking. The usage of asyncOSSFSis similar to the synchronous
variant; one simply needs to replaceOSSFileSystemwithAioOSSFileSystemneed to do is replacing theOSSFileSystemwith theAioOSSFileSystemimportossfsfs=ossfs.AioOSSFileSystem(endpoint='http://oss-cn-hangzhou.aliyuncs.com')print(fs.cat('/dvc-test-anonymous/LICENSE'))b'                                 Apache License\n'...Althoughaiooss2is not officially supported, there are still some
features that are currently lacking. However, in tests involving theput/getof 1200 small files, the async version of ossfs ran ten times
faster than the synchronous variant (depending on the pool size of the
concurrency).Tasktime cost in (seconds)put 1200 small files via OSSFileSystem35.2688 (13.53)put 1200 small files via AioOSSFileSystem2.6060 (1.0)get 1200 small files via OSSFileSystem32.9096 (12.63)get 1200 small files via AioOSSFileSystem3.3497 (1.29)ContributingContributions are very welcome.
To learn more, see theContributor Guide.LicenseDistributed under the terms of theApache 2.0 license,Ossfsis free and open source software.IssuesIf you encounter any problems,
pleasefile an issuealong with a detailed description.
Package: os-signpost
os-signpostmacOS provides theOSSignposterAPI
for measuring the duration of tasks in a program or marking events. These
so-called signposts can be visualized in theInstruments.appapplication.
This Python package provides a wrapper for theOSSignposterAPI, so that you
can mark tasks and events in Python applications.⏳ InstallMake sure you haveXcodeinstalled and
then install withpip:pipinstallos-signpost🚀 QuickstartCreating aSignposterSignposteris the main class of this package. You should create an instance ofSignposterin order to emit an event. You can create an instance as follows:fromos_signpostimportSignpostersignposter=Signposter("com.example.my_subsystem",Signposter.Category.DynamicTracing)The first argument to the constructor is the name of the subsystem, which is
usually the name of your organization followed by the name of the subsystem.
The second argument is the category within the subsystem. The system uses the
category to filter events. There are three predefined categories available:Category.PointsOfInterest: signposts with this category are always enabled.
ThePoints of Interestinstrument shows signposts with this category.Category.DynamicTracing: signposts with this category are only enabled when
recording with a performance tool like Instruments.app. Create an os_signpost
instrument that filters for the subsystem name to visualizeDynamicTracingsignposts.Category.DynamicStackTracing: this category behaves likeDynamicTracing,
but also records backtraces.Make signposts for the duration of a tasksTo measure the duration of a task, you can create signposts that mark an
interval. TheSignposter.begin_intervalmethod adds a signpost that marks the
beginning of an interval and returns a callback. This callback then marks the
end of the interval:end_interval=signposter.begin_interval("begin my task")# The task for which you want to measure the duration.end_interval("end my task")Both thebegin_intervalmethod and the callback take a message.
Instruments.app includes these messages in the visualization and allows you to
categorize intervals by their messages. When no message is specified for the
interval end, the same message as the begin will be used.You can also use theSignposter.use_intervalcontext manager in place of the
callback approach:withsignposter.use_interval("begin my task","end my task"):# The task for which you want to measure the duration.use_intervaltakes the begin and end messages as its arguments. The end
message is also optional for the context manager.Emitting an eventBesides marking intervals, you can also emit an event signpost. Instruments.app
shows events as single points. TheSignposter.emit_eventmethod emits an
event:signposter.emit_event("lp0 on fire")🙈 LimitationsThe macOS signpost API also allows associating a name with a signpost. However,
this name is required to be a C string literal. Therefore, we hardcode the name
of signpostspython.
Package: ossindex-lib
Python Library for quering OSS IndexThis OSSIndex module for Python provides a common interface to querying theOSS Index.This module is not designed for standalone use. If you're looking for a tool that can detect your application's dependencies
and assess them for vulnerabilities against the OSS Index, perhaps you should check outJake.You can of course use this library in your own applications.InstallationInstall from pypi.org as you would any other Python module:pip install ossindex-libUsageFirst create an instance ofOssIndex, optionally enabling local cachingo = OssIndex()Then supply aListofPackageURLobjects that you want to ask
OSS Index about. If you don't want to care about generating this list yourself, perhaps look to a tool likeJake(which uses this library) and will do all the hard work for you!As a quick test, you could run:o = OssIndex()
results: List[OssIndexComponent] = o.get_component_report(packages=[
    PackageURL.from_string(purl='pkg:pypi/pip@19.2.0')
])
for r in results:
    print("{}: {} known vulnerabilities".format(r.get_coordinates(), len(r.get_vulnerabilities())))
    v: Vulnerability
    for v in r.get_vulnerabilities():
        print('    - {}'.format(str(v)))... which would output something like ...pkg:pypi/pip@19.2.0: 1 known vulnerabilities
    - <Vulnerability id=e4c955a3-2004-472e-920b-783fea46c3cd, name=OSSINDEX-783f-ea46-c3cd, cvss_score=3.6>LoggingThis library send log events to a standard Pythonloggernamedossindex. You can configure the logger to output as
required through the standardPython logging configuration.TodosSupport authentication against OSS IndexPython SupportWe endeavour to support all functionality for allcurrent actively supported Python versions.
However, some features may not be possible/present in older Python versions due to their lack of support.ChangelogSee ourCHANGELOG.The Fine PrintRemember:It is worth noting that this isNOT SUPPORTEDby Sonatype, and is a contribution of ours to the open source
community (read: you!)Use this contribution at the risk tolerance that you haveDo NOT file Sonatype support tickets related toossindex-libDO file issues here on GitHub, so that the community can pitch inPhew, that was easier than I thought. Last but not least of all - have fun!
Package: ossit
Ossit API DocumentationOssit is an easy-to-use statistic collecting application that analyzes and displays your data. It follows a three-tier structure:StructureDomains: The highest level, which produces keys for user authentication. To set your Ossit domain key, import the Ossit library and set the domain key as follows:Statistic Groups: This tier allows you to group common statistics, providing better organization and analysis of your data. For example, you could create a group for "Website Traffic" that includes statistics such as page views, unique visitors, and bounce rate.Statistics: The lowest level, consisting of individual statistics with a name and value.Authentication and AuthorizationTo authenticate your requests, set your Ossit domain key. You can find your domain key in the Ossit dashboard.importossitossit.set_domain_key('YOUR_SECRET_KEY')ossit.set_time_zone('America/Toronto')Base URLThe base URL for the Ossit API ishttps://ossit.ca/v1/api. All API responses will be in JSON format.Endpoint StructureEach tier and future objects will follow the same URL path structure. For example, the Statistic Groups resource has the following endpoints:POST /statistic-groups: Creates a new statistic group.GET /statistic-groups: Retrieves a list of all statistic groups.PUT /statistic-groups/:id: Updates the specified statistic group.GET /statistic-groups/:id: Retrieves the specified statistic group.ExamplesHere's an example of creating a new statistic group using the Ossit package:importossitossit.domain_key='YOUR-KEY'group=ossit.StatisticGroup.create('Automations')print(group)To retrieve a list of all statistic groups:groups=ossit.StatisticGroup.list()print(groups)Error HandlingIf an error occurs, the API will return an error response in the following format:{"error":{"code":"ERROR_CODE","message":"Error description"}}Common error codes include:400 Bad Request: The request is malformed or contains invalid data.401 Unauthorized: The request is missing or has an invalid authentication token.403 Forbidden: The authenticated user does not have permission to perform the requested action.404 Not Found: The requested resource could not be found.500 Internal Server Error: An unexpected error occurred on the server side.Rate Limiting and ThrottlingTo ensure fair usage, the Ossit API enforces rate limiting. By default, users are limited to 1000 requests per hour. If you exceed this limit, you will receive a 429 Too Many Requests response.VersioningThe current version of the Ossit API is v1. To access different versions of the API (if available), replace the version number in the base URL (e.g.,https://ossit.ca/v2/api).ChangelogMaintain a changelog of API updates and improvements. Include the version, release date, and a brief summary of changes for each update.Support and Contact InformationFor API support and feedback, please contact our support team atwesleyh@stratusadv.com