Saravanan666/pypi-org huggingface dataset:
Package: shopcloud-django-instrumenting
Instrumentinginstallinstall with your favorite Python package managerpip3installshopcloud-django-instrumentingLogRequestMiddlewareadd additional Information from request in AppEngine to Logusageadd to MIDDLEWARE in django-app settings.py:MIDDLEWARE=[...'shopcloud_django_instrumenting.middleware.LogRequestMiddleware',]tracingfromshopcloud_django_instrumentingimporttracingtr=tracing.Tracer('name_of_service','name_of_operation')withtr.start_span('event.processing')asspan:passdata=tr.close()error managementA special layer on top of the API is the error boundary within the with blocks of spans. This layer catches exceptions and prints them. The pattern here is to catch all runtime errors and return a 422 status, indicating that an unexpected error has occurred, not a validation error. If you need to return validation data to the user, you must return a Result object on your own.@action(detail=True,methods=["post"],url_path="my-action")defmy_action(self,request,pk):tr=tracing.DjangoAPITracer(request)serializer=serializers.ProductSyncSerializer(data=request.data)serializer.is_valid(raise_exception=True)withtr.start_span('proceed')asspan:raiseException('something unhandle')trace_data=tr.close()returnResponse({'trace':trace_data,},status=status.HTTP_201_CREATEDiftr.is_successelsestatus.HTTP_422_UNPROCESSABLE_ENTITY)Another pattern to note is that everywithblock should contain a repeatable operation. You can create structures where you can run blocks side by side and only repeat some of them. However, be careful: while thewithblock catches exceptions, it may not guarantee that all variables are set.tr=tracing.DjangoAPITracer(request)is_a_success=Falseis_b_success=Falseis_c_success=Falsewithtr.start_span('A')asspan_a:is_a_success=Falsewithtr.start_span('B')asspan_b:is_b_success=Truewithtr.start_span('C')asspan_c:ifis_a_success:passdevelop$pytest
$pip3installcoverage# shell report$coveragerun-mpytest&&coveragereport--show-missing# html report$coveragerun-mpytest&&coveragehtml
$cdhtmlcov
$python3-mhttp.server
Package: shopcloud-django-monitoring
shopcloud-django-monitoringMonitoring AppQuickstartpip3 istall shopcloud-django-monitoringAdd "monitoring" to your INSTALLED_APPS setting like this::INSTALLED_APPS=[...'monitoring',]PLUGINS={'MONITORING_SOURCES':{'INSTALLED':['SQL_QUERY_V1','SQL_QUERY_V2','SQL_SAGE_GATEWAY_V1','NOT_SUCCESS_V1',]}}APP_TITLE=os.environ.get("APP_TITLE","Foobar-Services")Include the polls URLconf in your project urls.py like this::path('monitoring/', include('monitoring.urls')),Runpython manage.py migrateto create the polls models.Start the development server and visithttp://127.0.0.1:8000/admin/to create a metric (you'll need the Admin app enabled).Visithttp://127.0.0.1:8000/monitoring/to participate in the Monitoring Metrics.Queuemonitoring-metrics-proceedanlegen.Release$rm-rfbuilddist
$pip3installwheeltwine
$python3setup.pysdistbdist_wheel
$twineuploaddist/*
Package: shopcloud-eventbus
Shopcloud-EventBusAsync Processing, write to Log an Event and send it to a Queuegraph TD;
    AppEngine-->Log;
    Log-->EventBus;
    EventBus-->AppEngine;Usage$pipinstallshopcloud-eventbus$eventbusinit
$eventbusdeployVanilla:fromshopcloud_eventbusimportEventevent=Event(name="de.talk-point.platform/module/model/sync",model=self,)event.add_task(queue="default",url=f"module/api/model/{self.id}/action/",json={})event.fire()Djangouse for example theshopcloud-django-toolboxlibraryfromshopcloud_django_toolboximportEventevent=Event(name="de.talk-point.platform/module/model/sync",model=self,)event.add_task(queue="default",url=f"module/api/model/{self.id}/action/",json={})event.fire()InstallServices$ gcloud services enable cloudtasks.googleapis.com --project="test-eventbus-project"
$ gcloud pubsub topics create events --project='test-eventbus-project'
$ gcloud tasks queues create default --project='test-eventbus-project' --location='europe-west3'Log SinkCreate alog sinkwith the following filterresource.type="gae_app" "event-fire"$eventbus--help
$eventbusdeployDevelopment$python3setup.pyinstall
$python3-mshopcloud_eventbus-dinit--project="test-eventbus-project"--region="europe-west3"--app-endpoint="https://test-eventbus-project.ey.r.appspot.com"--secrethub-endpoint-user="talk-point/app-eventbus-test/production/evenbus-user"--secrethub-endpoint-pwd="talk-point/app-eventbus-test/production/evenbus-pwd"$python3-mshopcloud_eventbus-ddeploydann kann das Script getestet werden per$cd.eventbus
$python3-mvenvvenv
$sourcevenv/bin/activate
$pipinstall-rrequirements.txt
$functions-framework--targetmain_http--debugoder deployen und eine Test-Message durchsendenDeploy$rm-rfbuilddist
$pip3installwheeltwine
$python3setup.pysdistbdist_wheel
$twineuploaddist/*
Package: shopcloud-metric
Shopcloud Metric CLIinstall$ pip install shopcloud_metricUsageJob:Metrics wenn ein Job ausgeführt wirdfromshopcloud_metricimportJobMetricjob=JobMetric('sage-supplier-update',instance='airflow-production',env='testing',)withjobasj:j.gauge('job_last_success_unixtime',labels={'supplier':'EP'})raiseException('AS a')j.gauge('job_last_success_unixtime',labels={'supplier':'SP'})job.gauge('job_last_success_unixtime',labels={'supplier':'SP'})job.success()RawMetrics:fromshopcloud_metricimportMetricRegistryregistry=MetricRegistry()registry.gauge('job_up',value=1,labels={'name':'shopcloud-infrastructure-cli','instance':'tests'})registry.push()registry.gauge('job_up',value=0,labels={'name':'my-job','instance':'airflow-production'})registry.gauge('job_last_success_unixtime',labels=[{'name':'test'}])registry.gauge('api_http_requests_total',value=12,labels={'operation':'create'})registry.gauge('process_cpu_seconds_total',value=1)registry.gauge('http_request_duration_seconds',value=1)registry.push()Deploy to PyPi$pip3installwheeltwine
$python3setup.pysdistbdist_wheel
$twineuploaddist/*
Package: shopcloud-microservice
Microservice CLITool for manage the talk-point microservices.Install$pipinstallshopcloud-microserviceApp$microserviceapphealthSecurity$microservicesecuritymerge-security-pull-requests
$microservicesecurityci-secrets-rollingBackupBackup commands (test) (es wird ein Export pro Instanz ausgeführt)$python-mshopcloud_microservice-d-sbackupsql-initshopcloud-secrethub:europe-west3:secrethub# initialise backup storage for a instance$python-mshopcloud_microservice-d-sbackupsql-list-instances# list all known instances from projects.yaml$python-mshopcloud_microservice-d-sbackupsql-list-databasesshopcloud-secrethub:europe-west3:secrethub# list databases on server$python-mshopcloud_microservice-d-sbackupsql-dump# dumo all databaeses ob all known server from projects.yaml$python-mshopcloud_microservice-d-sbackupsql-dumoshopcloud-secrethub:europe-west3:secrethub# dump all databases from instance$python-mshopcloud_microservice-d-sbackupsql-dumpshopcloud-secrethub:europe-west3:secrethub:shopcloud-secrethub-api# download a database from instance$python-mshopcloud_microservice-d-sbackupsql-dumpshopcloud-secrethub:europe-west3:secrethub:shopcloud-secrethub-api# download a database from instance$python-mshopcloud_microservice-d-sbackupsql-download# download all dumpfiles to db-dumps$python-mshopcloud_microservice-d-sbackupsql-push-to-drive/tmp# sync content from download-folder to specific pathDevelopEntwickle deine Änderungen im develop branch oder feature branch.
Beim mergen des Pull-Requests in den master wird automatisch ein deploy mittelswheelundtwinedurchgeführt.
Package: shopcloud-secrethub
Shopcloud SecretHub CLIThe SecretHub CLI provides the command-line interface to interact with the SecretHub API.install$ pip install shopcloud_secrethubUsageReading and writing secrets:$secrethubauth
$secrethubread<secret-name>
$secrethubwrite<secret-name><value>Provisioning your applications with secrets:Provision a template file$secrethubinject-iapp.temp.yaml-oapp.yaml# app.temp.yamlenv_variables:ENV:{{talk-point/test-repo/env}}SECRET_KEY:{{talk-point/test-repo/secret_key}}Provision to the environment$eval`secrethubprintenv-iapp.temp.yaml`# app.temp.yamlenv_variables:ENV:{{talk-point/test-repo/env}}SECRET_KEY:{{talk-point/test-repo/secret_key}}in Code:fromshopcloud_secrethubimportSecretHubhub=SecretHub(user_app="test-script",api_token='<TOKEN>')hub.read('talk-point/test-repo/secret_key')Deploy to PyPi$rm-rfbuilddist
$pip3installwheeltwine
$python3setup.pysdistbdist_wheel
$twineuploaddist/*
Package: shopcloud-serverless
Serverless CLIServerless tool for the Google Cloud. You need thegcloudinstalled.Install$pipinstallshopcloud-serverless
$serverlessinitJobs$serverlessjobsinit
$serverlessjobscreate<job-name>
$serverlessjobsdeploy<job-name>
$serverlessjobsrun<job-name>Secrets:Secrets can you write in the.env.tempfile with the SecretHub syntax.GatewayThe main entrypoint for you serverless endpoint api is the gateway.Init the gateway with the function and then deploy the endpoints and then you can deploy the api.yaml file with gateway deploy endpoint.$serverlessgatewayinit$serverlessgatewaydeployEndpointsCreate a new endpoint for every path.$serverlessendpointsinit
$serverlessendpointscreatehealthAdd the Endpoint in theapi.yamltheoperation_idmust be unqie and is the identifier for the library.
You can change the<endpoint-name>.yamlwith the parametersmemory: memory in MBruntime: runtime of the function "python312"trigger: http or pubsub the value is the name of thetopicdependencies: as string arrayfor development in the background we usefunctions-framework$serverlessendpointsservehealthrun the integration test suite$serverlessendpointstesthealththen deploy the function$serverlessendpointsdeployhealth
Package: shopee-api-wrapper
Shopee API Wrapper - Not officialInstallpip install shopee-api-wrapperQuickstartFetch productfromshopee_apiimportShopeedefmain():# Configuring origin of endpointsshopee=Shopee(origin_url="https://shopee.com.br")# Fetching product by URLdata=shopee.fetch_product(url="https://shopee.com.br/Videogame-Stick-10mil-2-Controles-Sem-Fio-Console-Original-Portatil-Jogos-Retro-Antigo-Marisa-i.400311012.18265078100")# Printing data to the terminalprint(data)if__name__=="__main__":main()Detail: you don't need to do thisdef main(): ...;if __name__ == "__main__": main()structure, it's just good practice :)That's it for now, if you have any suggestions, don't hesitate to create an Issue
Package: shopee-crawler
Shopee-crawlerThis is a light crawling tool.High performance.Easy to use.Requirementpython>=3.7InstallUsingsetup.pygit clone https://github.com/lthoangg/shopee-crawler
python/python3 setup.py installUsingpippip install shopee-crawlerHow to useExamples:fromshopee_crawlerimportCrawlercrawler=Crawler()crawler.set_origin(origin="shopee.vn")# Input your root Shopee website of your country that you want to crawldata=crawler.crawl_by_shop_url(shop_url='shop_url')data=crawler.crawl_by_cat_url(cat_url='cat_url')data=crawler.crawl_by_search(keyword='keyword')data=crawler.crawl_cat_list()# print(data)UsageAbout 12.000 rows (products) in 2-3 seconds (32 workers)About 12.000 rows (products) in 5-6 seconds (16 workers)Test yourselfimporttimefromshopee_crawlerimportCrawlercrawler=Crawler()crawler.set_origin(origin="shopee.vn")keyword='điện thoại samsung'# Start timestart=time.time()# Crawldata=crawler.crawl_by_search(keyword=keyword)# End timeend=time.time()# Time crawlingprint("Time : ",end-start,"seconds")
Package: shopee-crawler-pkg-lthoangg
Shopee-crawlerThis is a crawl tool.Requirementpython>=3.7InstallUsingsetup.pygit clone https://github.com/lthoangg/shopee-crawler
python/python3 setup.py installUsingpippip install shopee-crawlerHow to useCrawl by shop urlfromshopee_crawlerimportcrawl_by_shop_urldata=crawl_by_shop_url('shop_url')# print(data)Crawl by category urlfromshopee_crawlerimportcrawl_by_cat_urldata=crawl_by_cat_url('cat_url')# print(data)
Package: shoper
shoperSimple shell operator module for PythonInstallation$pipinstall-UshoperExampleList directory contents withls.fromshoper.shelloperatorimportShellOperatorsh=ShellOperator()sh.run('ls -l')Write and sort random numbers.fromshoper.shelloperatorimportShellOperatorsh=ShellOperator()sh.run(args=['echo ${RANDOM}| tee random0.txt','echo ${RANDOM}| tee random1.txt','echo ${RANDOM}| tee random2.txt'],output_files_or_dirs=['random0.txt','random1.txt','random2.txt'])sh.run(args='sort random[012].txt | tee sorted.txt',input_files_or_dirs=['random0.txt','random1.txt','random2.txt'],output_files_or_dirs='sorted.txt')
Package: shopify-api
Shopify APIAPI client for theShopifyservice.ConfigurationNameTypeDefaultDescriptionSHOPIFY_API_VERSIONstrNoneThe Shopify API version to be used. Example:2023-01.SHOPIFY_API_KEYstrNoneThe username/key to be used in the authentication with the Shopify API.SHOPIFY_PASSWORDstrNoneThe password to be used in the authentication with the Shopify API.SHOPIFY_SECRETstrNoneThe shared secret to be used for message validation with the Shopify API.SHOPIFY_STOREstrNoneThe name/domain of the Shopify store to use the Shopify API.SHOPIFY_WEBSITEstrNoneThe public name/domain of the Shopify store to use the Shopify API, may be different fromSHOPIFY_STORE.SHOPIFY_IDstrNoneWhile using OAuth provides a way to define the client identifier.SHOPIFY_SECRETstrNoneWhile using OAuth provides a way to define the client secret.SHOPIFY_REDIRECT_URLstrNoneTo be used in the OAuth process as the target redirect URL.LicenseShopify API is currently licensed under theApache License, Version 2.0.Build Automation
Package: shopify-python-api
Because of its convenience and simplicity, many online stores are built on Shopify. This means that these stores have a similar structure, which facilitates the scraping process.The Shopify API Python library provides a convenient way to interact with the Shopify Collections API and the Shopify Products API. Using the provided API key. This library lets you get information about collections and products from any Shopify store.FeaturesOur Python Shopify API has some features and benefits:Retrieve information about collections in a Shopify store.Fetch detailed product information from a Shopify store.Easily integrate the library into your Python scripts or projects.Installpip install shopify-python-apiAPI keyTo access the Shopify API Python library, you need an API key from Scrape-It.Cloud. Here's how you can obtain the API key:Go to theScrape-It.Cloudwebsite in your web browser.Sign up for a new account by clicking the "Sign Up" button.You will also receive some free credits that you can use for API requests. The available credits will be displayed on the dashboard.After that, you can use your API key in your scrapers.UseTo start using the Shopify API library, use this example:fromshopify_apiimportShopifyAPIshopify_api_client=ShopifyAPI("INSERT_YOUR_API_KEY")try:result=shopify_api_client.collections(params={"url":"https://libertasbella.com/"})print(result)exceptExceptionase:print(e.validation_errors)Using this example, you can get Shopify collections. The function has only one required parameter:url (required): The URL of the Shopify store.You can also use our Shopify Python API to get information about products:fromshopify_apiimportShopifyAPIshopify_api_client=ShopifyAPI("INSERT_YOUR_API_KEY")try:result=shopify_api_client.products(params={"url":"https://libertasbella.com/","limit":250,"page":1})print(result)exceptExceptionase:print(e.validation_errors)Here, in addition to the required parameter, you can also use optional ones to get exactly the data you need:limit (optional): The maximum number of products to retrieve. It must be a number between 1 and 250.page (optional): The page number of the results to retrieve. It must be a number starting from 1.collection (optional): The collection handle to filter the products.For more detailed information about the response format and parameters, you can go to thedocumentation page.How to use Shopify API in Python for ScrapingTo use the Shopify API Python library, install it and import it into your Python script.Use the library functions to get the information you need. For example, you can get collections and products from Shopify by calling the appropriate functions provided by the library.Remember to replace "YOUR_API_KEY" with the API key you received from Scrape-It.Cloud. Following these steps, you can use the Shopify API Python library to retrieve data from Shopify stores and integrate it into your apps or projects.You can use ourShopify no-code scraperif you don't want to write a script and just need to get information. It allows you to retrieve data from Shopify stores without writing any code.
Package: shopify-requests
Shopify RequestsShopifyRequests is an API client for Shopify built on top of therequestslibrary.The main goals of this project are:remove boiler plate code needed to do a single API callwork well with testing libraries like VCR and RequestsMockeasy to useThere are also future plans to supporthttp2,asyncio, andGraphQL.Usagefromshopify_requestsimportRestClientclient=RestClient('foo.myshopify.com',access_token='abc123')response=client.get('shop.json')TheRestClientis the configuration point so that all requests made with it will have the same options.
Some of the options you can configure are:Oauth token vs private app tokenAPI versionSafe retriesRate limit backoffFor more configuration options check out theAPI DocsAn additional benefits of using the same client is that it will reuse the same TCP connection so the SSL handshake only has to happen once.Did you find a bug or have a question?Theissue boardwill be the best place to reach out and get the problem sorted out.
Package: shopify-scrape
shopify-scrapeInstallationpip install shopify_scrapeUsageExtracts json data for given URL.python -m shopify_scrape.extract url -husage: extract.py url [-h] [-d DEST_PATH] [-p PAGE_RANGE [PAGE_RANGE ...]]
                      [-c] [-f FILE_PATH]
                      url

positional arguments:
  url                   URL to extract. An attempt will be made to fix
                        inproperly formatted URLs.

optional arguments:
  -h, --help            show this help message and exit
  -d DEST_PATH, --dest_path DEST_PATH
                        Destination folder for extracted files. If
                        subdirectories present, they will be created if they
                        do not exist. Defaults to current directory './'
  -p PAGE_RANGE [PAGE_RANGE ...], --page_range PAGE_RANGE [PAGE_RANGE ...]
                        Inclusive page range as tuple to extract. There are 30
                        items per page. If not provided, all pages with
                        products will be taken.
  -c, --collections     If true, extracts '/collections.json' instead of
                        '/products.json'
  -f FILE_PATH, --file_path FILE_PATH
                        File path to write. Defaults to
                        '[dest_path]/[url].products' or
                        '[dest_path]/[url].collections'Extracts json data for URLs given in a specified column of a csv file.python -m shopify_scrape.extract batch -husage: extract.py batch [-h] [-d DEST_PATH] [-p PAGE_RANGE [PAGE_RANGE ...]]
                        [-c] [-r ROW_RANGE [ROW_RANGE ...]] [-l [LOG]]
                        urls_file_path url_column

positional arguments:
  urls_file_path        File path of csv file containing URLs to extract.
  url_column            Name of unique column with URLs.

optional arguments:
  -h, --help            show this help message and exit
  -d DEST_PATH, --dest_path DEST_PATH
                        Destination folder for extracted files. If
                        subdirectories present, they will be created if they
                        do not exist. Defaults to current directory './'
  -p PAGE_RANGE [PAGE_RANGE ...], --page_range PAGE_RANGE [PAGE_RANGE ...]
                        Inclusive page range as tuple to extract. There are 30
                        items per page. If not provided, all pages with
                        products will be taken.
  -c, --collections     If true, extracts '/collections.json' instead of
                        '/products.json'
  -r ROW_RANGE [ROW_RANGE ...], --row_range ROW_RANGE [ROW_RANGE ...]
                        Inclusive row range specified as two integers. Should
                        be positive, with second argument greater or equal
                        than first.
  -l [LOG], --log [LOG]
                        File path of log file. If none, the log file is named
                        logs/[unix_time_in_seconds]_log.csv. 'logs' folder
                        created if it does not exist.
Package: shopify-stream-sdk
This module wraps the Ayinope Shopify Monitor websockets stream.InstallationThis module is available via pip:$ pip install shopify-stream-sdkBasic UsagefromshopifystreamimportShopifyStreamshopify=ShopifyStream()domain='www.example.com'# Optionalinclude=['include','these','keywords']# Optionalexclude=['exclude','some','others']# Optionalproduct=shopify.wait_for_product(domain=domain,include=include,exclude=exclude)print(product)
Package: shopifywebhookparser
Shopify Webhook ParserThe Shopify Webhook Parser is a versatile Python module designed for efficient parsing of Shopify webhook requests. While it includes a specific implementation for Azure Functions, its architecture is flexible, allowing it to be extended and adapted to various environments and requirements.FeaturesStrategy Pattern:At the core of this module lies the Strategy pattern, enabling dynamic selection and interchangeability of parsing logic. This approach allows for the easy addition of new parsing strategies to accommodate different processing needs or webhook formats.Extendable for Various Environments:Initially provided with an implementation for Azure Functions, the module is structured to be easily extendable to other platforms or frameworks.Robust Error Handling:Incorporates comprehensive error handling to manage and log parsing exceptions, ensuring reliability and maintainability.Customizable Parsing Strategies:Designed to cater to a range of webhook parsing needs, from simple to complex, the module is both customizable and scalable, fitting into different project sizes and complexities.Implementing New Parsing StrategiesTo integrate new parsing strategies, follow these guidelines:Strategy Function Signature:Implement strategies as callables conforming to the following type:(HttpRequest) -> Tuple[bytes, Dict[str, Any]].These functions should accept a webhook request and return a tuple with the request body in bytes and headers.Error Management:Strategies should handle and log exceptions effectively.Raise aValueErrorwith a descriptive message in case of processing failure.Seamless Integration:Design strategies for easy integration with theparse_shopify_webhook_requestfunction.Ensure compatibility with expected input types.Comprehensive Testing:Thoroughly test each strategy to accurately parse various webhook formats.Include tests for edge cases and error handling scenarios.InstallationpipinstallshopifywebhookparserUsageThis module can be used in Azure Functions to parse Shopify webhook requests. TheParsedWebhookclass is designed to be flexible, allowing direct instantiation with specific attributes or deriving them from provided webhook data.Exampleimportazure.functionsasfuncfromshopifywebhookparserimportparse_shopify_webhook_request,azure_func_request_parse_strategydefmain(req:func.HttpRequest)->func.HttpResponse:try:parsed_webhook=parse_shopify_webhook_request(req,azure_func_request_parse_strategy)# Access parsed dataonlinestore_name=parsed_webhook.onlinestore_name# Further processing of the parsed datareturnfunc.HttpResponse(f"Processed webhook for store:{onlinestore_name}",status_code=200)exceptValueErrorase:returnfunc.HttpResponse(f"Error:{e}",status_code=400)Development and ContributionsFeel free to contribute to the improvement of this module by submitting pull requests or reporting issues.LoggingThe module uses Python's built-in logging to provide insights into its operations and error conditions.LicenseMIT, for details see LICENSE