Saravanan666/pypi-org huggingface dataset:
Package: gcp-googleanalytics
google-analyticstakes the pain out of working with the Google
Analytics reporting APIs. It supports both the Core and the Real Time
API. It is written in Python but there’s also a full-featured
command-line interface.Authentication.OAuth2 is a bit of a pain and we’ve made it
easier, both for interactive use and forserver
applications.
We can also save your credentials in the operating system’s keychain
if you want to.Querying.Easier to query per week, month or year. Query using
metric IDs or using their full names, whichever you think is nicer.
Work with both the Core and the Real Time APIs.Reporting.Generate reports from the command-line. Optionally
describe reports and queries ineasy-to-read and easy-to-write YAML
files.
Reports in Python work better too: iterate through the entire report
or column-per-column.Exploration.Traverse the account hierarchy from account to
webproperty to profile to dimensions, both programmatically and with
the included command-line interface.Exports.Clean JSON and CSV – as well asPandasdata frames – so you can
analyze the data in anything from Excel to R.This package is built on top ofGoogle’s own API client for
Python.QuickstartFirst, install the package usingpipPython 2:pip install googleanalyticsPython 3:pip3 install googleanalyticsThen, create a new project in theGoogle Developers
Console, enable the Analytics
API under “APIs & auth > APIs” and generate credentials for an installed
application under “APIs & auth > Credentials”. Keep this page open. For
more detail, look at thewiki page on
authentication.After that, executing your first query is as easy asimportgoogleanalyticsasgaaccounts=ga.authenticate()profile=accounts[0].webproperties[0].profilepageviews=profile.core.query.metrics('pageviews').range('yesterday').valueprint(pageviews)Or on the command-line, that’d be:googleanalytics--identity<identity>--account<account>--webproperty<webproperty>\querypageviews--startyesterdayThe account, webproperty and profile determine what data you’ll be
querying. Learn more about profiles and querying on theQueryingwiki page, or alternatively look at theCommon
Queriespage for lots of examples. Read more about how to work with the
resulting data inWorking With
Reports.On The
Command-Linehas more details about the command-line application.The example above will authorize the app and authenticate you
interactively. It is also possible to pass credentials as arguments in
Python, using environment variables or from your operating system’s
keychain. Authentication is treated in much more depth on theauthentication wiki
page.
Package: gcp-hashicorp-packer-reaper
AWS Hashicorp Packer ReaperHashicorp Packer is a great tool for building Google Compute Engine Images. However, sometimes the virtual machine running packer
is not stopped. This utility stops or terminated all virtual machines with the name tag 'Packer Builder'You can use it as a command line utility or install it as an Cloud Run service and stop the spend , NoOps style!install the packer reaperto install the packer reaper, type:pipinstallgcp-hashicorp-packer-reapershow running packer instancesTo show running packer instances:$gcp-hashicorp-packer-reaperlist
packer-5e99d4f2-b5a5-e9e0-b763-cd5102ae7e73launched7minutesagoinyour-project-europe-west4-c-RUNNING
INFO:1packerbuilderinstancesfoundstop running packer instancesTo stop running packer instances older than 2 hours:$gcp-hashicorp-packer-reaperstop--older-than2h

INFO:stoppingpacker-5e99d4f2-b5a5-e9e0-b763-cd5102ae7e73inyour-projectcreated3hoursago
INFO:totalof1runninginstancesstoppeddelete running packer instancesTo terminate stopped and running packer instances older than 24 hours:gcp-hashicorp-packer-reaper--verbosedelete--older-than24h

INFO:deletingpacker-5e99d4f2-b5a5-e9e0-b763-cd5102ae7e73inyour-projectcreated2daysago
INFO:totalof1instancesdeleteddeploy the packer reaperTo deploy the packer reaper as a service in your project, type:gitclonehttps://github.com/binxio/gcp-hashicorp-packer-reaper.gitcdgcp-hashicorp-packer-reaperPROJECT=$(gcloudconfigget-valueproject)makeUSERNAME=$PROJECTsnapshotcdterraform
terraforminit
terraformapply-varproject=$PROJECT-auto-approveThis will install the packer reaper in your GCP project and run every 10 minutes, and delete instances older than 2 hours. You can change the schedule and the action to meet your requirements.
Package: gcp-impersonation-wrapper
GCP Impersonation wrapperThe gcp-impersonation-wrapper is a small util used to impersonate service accounts using service account. We pass the main credentials, list service accounts we can impersonate and then actually impersonate like so:#!/usr/bin/env python3fromsrc.service_accountimportServiceAccountif__name__=="__main__":# Insert credentials herecredentials=Noneservice_account=ServiceAccount(project_id="your-project-id",impersonator_credentials=credentials)# Can be used in a with statementwithservice_account.impersonate(service_account_email="your-service-account-email",target_scopes=["https://www.googleapis.com/auth/cloud-platform"],service="iam",version="v1")asimpersonated_service_account:# do something with the impersonated service accountservice_accounts_the_impersonated_sacc_can_see=impersonated_service_account.service.projects().serviceAccounts().list().execute()print(service_accounts_the_impersonated_sacc_can_see)# Or as a variableimpersonated_service_account=service_account.impersonate(service_account_email="your-service-account-email",target_scopes=["https://www.googleapis.com/auth/cloud-platform"],service="iam",version="v1")service_accounts_the_impersonated_sacc_can_see=impersonated_service_account.service.projects().serviceAccounts().list().execute()To list the impersonatable service accounts we do:#!/usr/bin/env python3fromsrc.service_accountimportServiceAccountif__name__=="__main__":# Insert credentials herecredentials=Noneservice_account=ServiceAccount(project_id="your-project-id",impersonator_credentials=credentials)# This returns a list of email like ["john@doe.com", "jane@doe.com", ....]service_account_emails_we_can_impersonate=service_account.list_all_impersonatable_service_account_emails()
Package: gcp-io
GCP-IOGenerate PyTorch datasets from files stored in Google cloud storage and Google drive.InstallationRun the following to install this package:pipinstallgcpioEnvironment Variables RequiredLOG_FILE_PATH: Path to store the logs in a log file
LOG_LEVEL: LOG LEVEL(Example: INFO, DEBUG, ERROR etc)
TOKEN_PATH_GDRIVE: path to the generated token.json fileExample usage:Google Drive:fromgcpio.gdriveimportGdrive# Get meta data of files present in a Google Drive folderGDRIVE=Gdrive()files=GDRIVE.get_files_metadata(folder_id="XXXXX",page_size=500,file_type="image/png",replace_query=None)# returns a dict with keys['files','len'] where files is a list of objects from the Drive folder# Create the torch dataset from filesGDRIVE=Gdrive()dataset=g.create_dataset(data_folder_id=folder_id,labels_folder_id=folder_id,page_size=1000,data_file_type="image/png",labels_file_type="text/csv",skip_labels=skip_labels,)# load this dataset into torch dataloader.dataloader=DataLoader(dataset,batch_size,sampler=BatchSampler(SequentialSampler(dataset),batch_size=batch_size,drop_last=True),)#get a sample batchsamples=next(iter(dataloader))images=[]foriinrange(len(samples)):# ignoring labels and collecting images only for verificationimg,_=samples[i]images.append(img)images=torch.cat(images)grid=make_grid(images,nrow=20)save_image(grid,f"./examples/generated_images.png")outputs
Package: gcp-jupyterlab-shared
Common frontend components, server extension, and packages for GCP JupyterLab extensionsCommon package for shared frontend components and Python server modules
to be used across JupyterLab extensions.FeaturesThis module exposes two routes:/gcp/v1/metadata- A subset of theGCE VM Metadatawith AI Platform Notebook-specific keys pulled to the top-level to identify
the ML framework installed on the machine./gcp/v1/proxy- A proxy-endpoint that accepts a base64-encoded Google API
endpoint as a path and forwards the request to the decoded API service method.
This is provided so that frontend code can issue API requests to the
JupyterLab server, which will attach its machine credential to the request
before forwarding it to the API service. This avoids the problem of requring
end-user authentication at the UI, or passing the machine credential to the
browser which is not supported.DevelopmentFollow theinstructionsfrom the root of the repository.Install this package in editable mode bycd-ing into this directory
and running the../dev-install.shscript.When developing locally, in order to access a metadata server running on GCE VM,
you will need to issue the following command to set up port forwarding.
This will allow requests made locally to port 8889 to be forwarded to a real GCE
metadata server. Do this in another terminal window since it will create an
interactive SSH session.gcloud compute ssh ${INSTANCE} --ssh-flag "-L 8889:metadata.google.internal:80"Set the METADATA_SERVER environment variable to tell the server to use
the forwarded addressexport METADATA_SERVER="http://localhost:8889"
Package: gcp-jwt
GCP-JWTGCP-JWT is a library to create and sign tokens using the Google Cloud
Platform's Key Management Service. It handles the signing and verification
of the token.Key FeaturesToken SignageSignature VerificationExpiry VerificationAuto Rotating Asymmetric KeyBatch Token GenerationCompression OptionSymmetrically Encrypted LayerPretty APIGCP Roles NeededTo be able to use this library you'll need a GCP service account with at
least the following roles:cloudkms.cryptoKeyVersions.listcloudkms.cryptoKeyVersions.useToSigncloudkms.cryptoKeyVersions.viewPublicKeyWhy not using GCP integrated auth ?Some projects require more flexibility, I've also personally came
across a case where a company didn't want Google to manage the
authentication process. Whether or not this should be used in production
is outside the scope of this project.Simple Usagefromgoogle.cloudimportkmsfromgcpjwt.jwtimportJWTfromgcpjwt.jwt_signerimportJWTSigner# Create a client using a json file and initialise an asymmetric signer.client=kms.KeyManagementServiceClient.from_service_account_file('../resources/google.json')signer=JWTSigner(client,'your-project','your-ring-location')# Create a simple JWT without changing any claims.jwt=JWT(signer,'your-ring','your-key')text_token=jwt.token()Installpip install gcp-jwtNote:This project uses the cryptography package which needs to install natives. It
might not be compatible with App Engine Standard because of the limited libraries allowed.
Package: gcp-log
gcp-logPython library to format logs asGCP-compatible JSON.Features:Works with standardlogging.Supportsextrafor structured logging.100% compatible withthe official spec.Non-invasive, doesn't start any threads or processes, doesn't make network or system calls.Zero configuration.Fast.Pure Python.Type-safe.Zero dependency.Optional integration withorjsonfor better performance.Installationpython3-mpipinstallgcp-logOptional:If you also installorjson, gcp-log will automatically use it instead of stdlibjsonfor faster serialization:python3-mpipinstallorjsonUsageTo use it, simply setgcp_log.Formatteras theformatterfor the logger:importloggingimportgcp_loglogger=logging.getLogger()handler=logging.StreamHandler()formatter=gcp_log.Formatter()handler.setFormatter(formatter)logger.addHandler(handler)And then you can use the logger:logger.info('aragorn',extra=dict(father='arathorn'))The output:{"message":"aragorn","severity":"INFO","timestamp":"2022-03-17T10:09:58.393124+00:00Z","logging.googleapis.com/sourceLocation":{"file":"/full/path/to/example.py.py","line":24,"function":"test_gcp_formatter"},"father":"arathorn"}A human-readable version:{"message":"aragorn","severity":"INFO","timestamp":"2022-03-17T10:09:58.393124+00:00Z","logging.googleapis.com/sourceLocation":{"file":"/full/path/to/example.py.py","line":24,"function":"test_gcp_formatter",},"father":"arathorn",}
Package: gcpmate
GCPMate - Google Cloud Platform AssistantGCPMate is a tool that helps you manage your Google Cloud Platform (GCP) resources. It uses OpenAI's powerful language models to understand what you want to do, and then generates GCP command-line instructions for you to execute. If you like the proposed outcome, GCPMate can also execute the commands for you!InstallationTo use GCPMate, you'll need to have Python 3.x and an OpenAI API key. To install GCPMate, simply run:pipinstallgcpmateIn current version program expects to find API key inenvvariableOPENAI_API_KEY. You can obtain the API key athttps://platform.openai.com/account/api-keys. Once ready, you can set upenvvariable by running:exportOPENAI_API_KEY=<your-api-key>UsageGCPMate will use OpenAI's language models to understand your query, and then generate a series of GCP command-line instructions for you to execute. If you like the proposed outcome, GCPMate can also execute the commands for you! To use GCPMate, simply run:gcpmate"<your query>"Where<your query>is a description of what you want to achieve in GCP.To get an explanation for a command, error or even a custom query, you can use the following command:gcpmate--explain"your query"Replace "your query" with the text you want to explain. Be sure to use single quotes if your query contains double quotes, and vice versa.ExamplesHere are some example queries you can use with GCPMate:gcpmate"create a new GCP project called my-superb-new-project"gcpmate"create ubuntu VM called superbvm in us-central1-a in project <xyz>"You can also use GCPMate to ask queries, or paste error messages, or ask to explain a command:gcpmate--explain"Why I cannot connect to my VM over ssh?"gcpmate-e" gcloud compute instances create newvm --project superproject324 --zone us-central1-a --image-family ubuntu-1804-lts --image-project ubuntu-os-cloud"# this query will return explanation what that command would do.OptionsGCPMate supports several command-line options:gcpmate-h# Show the help message and exitgcpmate-m<model># Specify the OpenAI model to use (default: gpt-3.5-turbo)gcpmate-s# Skip printing "Fair warning" and runtime info (gcloud account, project, region, zone, OpenAI model)gcpmate-e# Return explanation to given query, which can be command, error message, etc.ContributingIf you find a bug or have a feature request, please open an issue on the GCPMate GitHub repository:https://github.com/ontaptom/gcpmatePull requests are also welcome! If you want to contribute to GCPMate, please fork the repository, create a new branch, and then submit a pull request. Please ensure that your code passes the existing tests and linting rules.LicenseThis project is licensed under the terms of the Apache License 2.0. See theLICENSEfile for more details.
Package: gcp-ng-helpers
Overviewgcp-ng-helpersis a Python library that provides helper functions for
interacting with Google Cloud Platform (GCP) services.InstallationUse the package manager pip to install gcp-ng-helperspip install gcp-ng-helpersFunctions submoduleHttp RouterCloud function by default only serves single '/' (root) http endpoint.
With this Http Router you can easily serve much more, without need to parse
Request object for method and pathUsageDefine a function with flask.Request argument
returning the flaks.Response object
Import HttpRouter from functions submodule and register defined function
as route with path and method
Then inside cloud function entry point function return router.responsefromflaskimportRequest,Response,make_responsefromgcp_helpers.functions.routersimportHttpRouterdefhello_route(request:Request)->Response:returnmake_response('Hello',200)router=HttpRouter()router.register(hello_route,'/hello','GET')defmain(request):returnrouter.response(request)orfromflaskimportRequest,Response,make_responsefromgcp_helpers.functions.routersimportHttpRouterrouter=HttpRouter()@router.route("/hello",'GET')defhello_route(request:Request)->Response:returnmake_response('Hello',200)defmain(request):returnrouter.response(request)Tasks submoduleManagerCloudTasksManager class is wrapper for cloud task api. Allows to shorten task creation.
Currently, supports only http task creationFirestoreCollectionFirestoreCollection and FirestoreCollectionGroup are wrappers for firestore read/write operations
bounded to the defined collection or collection group.
Package: gcp-oidc
gcp-oidcGenerate OpenID Connect tokens for GCP service accountsAboutThis library is a wrapper around the google-cloud-iam client library. It's primary function is to support automatically refreshing an expired id token.How it WorksTwo identities are involved in generating an id token:
the caller, and the client service account for whom the credential is created. The caller is responsible for requesting a token from Google for the client service account.The caller is a Google service account that's derived from theGOOGLE_APPLICATION_CREDENTIALSenvironment variable. Alternatively, the caller can be explicitly set through the client constructor by specifying a path to a service account file.The client service account (for whom the credential is created) is automatically set to the caller service account if not explicitly provided.The caller must have theserviceAccountTokenCreatorrole on the client service account in order to generate an id token for that account.ExamplesInstall the packagepip install gcp-oidcGenerate an id token for the default service account.from oidcutils import GCPOIDCClient

# Instantiate a OIDC client object and generate a token.
# Hang on to the client instance for future invocations.
# Any subsequent calls to get_id_token() will automatically 
# refresh the token if the token has expired.
client = GCPOIDCClient()
token = client.get_id_token()Generate an id token for a specific client service account when you know the client service account email address.from oidcutils import GCPOIDCClient

options = {
    'debug': True,
    'client_service_account_email': 'my-service-123@mygcproject.iam.gserviceaccount.com'
}
client = GCPOIDCClient(options)
token = client.get_id_token()Generate an id token for a specific client service account when you have a client service account JSON key file.from oidcutils import GCPOIDCClient

options = {
    'debug': True,
    'client_service_account_file': '/path_to_service_account_file.json'
}
client = GCPOIDCClient(options)
token = client.get_id_token()Generate an id token for a specific client service account using a specific caller.from oidcutils import GCPOIDCClient

options = {
    'debug': True,
    'caller_service_account_file': '/path_to_caller_service_account_file.json'
    'client_service_account_email': 'my-service-123@mygcproject.iam.gserviceaccount.com'
}
client = GCPOIDCClient(options)
token = client.get_id_token()Client Methods and PropertiesPropertyDescriptionexpiryDate time when the token expiresaudienceThe configured JWT audience claimclient_emailThe client email address--MethodDescriptionrefresh_id_token(options)Forces a token refresh. Takes an optional dict positional parameter. 'service_account_id' and 'audience' keys can be provided values. Return nothing.get_id_token()Returns a valid id token
Package: gcp-pubsub-dao
GCP Pub/Sub DAOThe library provides DAO classes for GCP pubsub publisher/subscriber.Installationpipinstallgcp-pubsub-daoUsagesync subscriber:fromgcp_pubsub_daoimportPubSubSubscriberDAO,Messagedao=PubSubSubscriberDAO(project_id="prodect-dev",subscription_id="subscription")messages:Message=dao.get_messages(messages_count=2)formessageinmessages:print(message.data)dao.ack_messages(ack_ids=[message[0].ack_id])dao.nack_messages(ack_ids=[message[1].ack_id])dao.close()# to clean up connectionssync publisher:fromgcp_pubsub_daoimportPubSubPublisherDAOdao=PubSubPublisherDAO(project_id="prodect-dev")try:dao.publish_message(topic_name="topic",payload=b"asdfsdf",attributes={"kitId":"AW12345678"})exceptExceptionasex:print(ex)async subscriber:fromgcp_pubsub_daoimportAsyncPubSubSubscriberDAO,Messagedao=AsyncPubSubSubscriberDAO(project_id="prodect-dev",subscription_id="subscription")messages:Message=awaitdao.get_messages(messages_count=2)formessageinmessages:print(message.data)awaitdao.ack_messages(ack_ids=[message[0].ack_id])awaitdao.nack_messages(ack_ids=[message[1].ack_id])async publisher:fromgcp_pubsub_daoimportAsyncPubSubPublisherDAOdao=AsyncPubSubPublisherDAO(project_id="prodect-dev")try:awaitdao.publish_message(topic_name="topic",payload=b"asdfsdf",attributes={"kitId":"AW12345678"})exceptExceptionasex:print(ex)async workerimportasyncioimportsyssys.path.append("./")fromgcp_pubsub_daoimportAsyncPubSubSubscriberDAOfromgcp_pubsub_dao.worker_poolimportWorkerPool,WorkerTask,HandlerResultfromgcp_pubsub_dao.entitiesimportMessageasyncdefhandler1(message:Message):print(f"handler1:{message}")awaitasyncio.sleep(2)returnHandlerResult(ack_id=message.ack_id,is_success=True)asyncdefhandler2(message:Message):print(f"handler2:{message}")awaitasyncio.sleep(5)returnHandlerResult(ack_id=message.ack_id,is_success=True)asyncdefmain():tasks=[WorkerTask(subscriber_dao=AsyncPubSubSubscriberDAO(project_id="ash-dev-273120",subscription_id="http-sender-sub"),handler=handler1,),WorkerTask(subscriber_dao=AsyncPubSubSubscriberDAO(project_id="ash-dev-273120",subscription_id="email-sender-sub"),handler=handler2,),]wp=WorkerPool()awaitwp.run(tasks=tasks)if__name__=="__main__":asyncio.run(main())
Package: gcp-python-client-functions
GCP Python Client FunctionsInstallationLaunch a terminal and install by entering:condainstallgcp_python_client_functions…orpipinstallgcp_python_client_functionsUsage Prerequisites1. GCP environmentCreate a GCP project by going to theResource
Managerpage in the cloud console (cloud resources naming
convention).Enable billing for the project you’ll work with by going to theCloud
Billingpage and selecting
your project on the top right corner (300 USD free credits are
associated with new billing accounts + abig list of free tier
products).Enable BigQuery API by going toAPIs &
Servicespage on the cloud
console and looking for the BigQuery API.2. AuthenticationAuthenticatingby… - (locally) using theGoogle Cloud
SDK(installation needed). Launch a
terminal enter:gcloudauthlogin--update-adc(elsewhere)downloading a service account JSON
keyfileand point to it using an environment variable:exportGOOGLE_APPLICATION_CREDENTIALS="/path/to/keyfile.json"or throughWorkflow Identity
Provider(what we used to authenticate within GitHub Actions, seeCloud
Advocate’s videoandstackoverflow
questionfor additional support).How to usefromgcp_python_client_functions.clientsimport*PROJECT_ID='dz-learning-d'# BigQuerybq_obj=BigQuery(PROJECT_ID)# Cloud Storagestg_obj=Storage(PROJECT_ID)