Saravanan666/pypi-org huggingface dataset:
Package: gcp-python-logging
gcp-python-loggingA python module for the handy printing of logs to StackDriver based on severity from within a Google Cloud FunctionAfter running into an issue with Python logging not logging to the correct StackDriver severity filter, I created this helper module in order to facilitate that function. It replicates much of the logging functionality, but translated into StackDriver levels.Severity LevelsSeverityInt LevelALL0DEBUG100INFO200WARNING400ERROR500CRITICAL600for all severity levels exceptALL, only logs of that log level or higher will be printed out (ex: Log Level ofWARNINGonly prints outWARNING,ERROR, andCRITICAL). If theALLlog level is written, it will appear in all cases.Setup the ModuleThis module can be used simply by doing:pip install gcp-python-loggingThis module can then be imported and initialized as such:from gcp_python_logging import LoggingClient

logger = LoggingClient()Passing no arguments will use the environment variables available to grab the project, function name, and function region. However you can also pass those variables into the client in order to change where these logs will be sent.Creating a LogIn order to create a new log, simply use a similar syntax to Python logger.logger.debug('This is a debug statement')
logger.error('Something went wrong!')Changing Log LevelBy default, this module sets the LogLevel toDEBUG, but this can be changed by running the command:logger.setLogLevel('ERROR')Then only logs at levelERRORor higher will be reported.Author's Note:This currently prints out all logs in a JsonPayload in order to meet the needs of my project. If users would prefer the ability to choose the printout type please feel free to reach out to me.
Package: gcp-saving
GCP saving package is implemented for deploying a function that you can invoke for stopping, deleting or starting each … instance.The goal is to implement this package for each GCP service can stop, delete and start its instances.It is part of theeducational repositoriesto learn how to write stardard code and common uses of the TDD.PrerequisitesYou can useServerless frameworkfor deploying the function:
if you want to use the guide below, you have to install Serverless framework before.If you want to use another GCP tool, you can see the repositorygcp-tool-comparisonbefore to implement your version.InstallationComing soon.Change LogSeeCHANGELOG.mdfor details.LicenseThis package is released under the MIT license.  SeeLICENSEfor details.
Package: gcpsecrets
gcpsecretsGCP Secret Manager as Python DictonaryInstallpip install git+https://github.com/tvaroska/gcpsecretsUssageDictionary accepts two types of keys:str: the latest active version of the sectertuple[str, str]: the exact version of the secretExamples:from gcpsecrets import GCPSecretssecrets = GCPSecrets() # to use other than default project use argument project=...api_key = secrets['API_KEY']
Package: gcp-secrets-kube
# Why gcp-secrets-kube?Store your secrets in [GCP secret manager](https://cloud.google.com/secret-manager) and build your
manifests for your favourite gitops tool.# Features
- Transform secrets from GCP secret manager to k8s secrets manifests
- Assign k8s namespaces according to dedicated secrets labels
- One to many relationship between GCP secrets and k8s secrets (with namespaces).Don’t repeat yourself!# Installpip install gcp-secrets-kube# Run
Get a secrets.yaml file by running:gcp-secrets-kube PROJECT_ID SECRET_ID  # latest secret versionorgcp-secrets-kube PROJECT_ID SECRET_ID –version 123# Miscellaneous
Build secrets in GCP and add labels likek8s-ns-xxxwith the namespace as the value. xxx could be 1, 2 or 3… according to how many namespaces you want the secret to be configured in.# Example of gitops approach with sealed-secrets
Gitops offers observability and control on your Kubernetes cluster, but secrets cannot be saved in
the git repository. Withgcp-secrets-kubeextract your secrets from gcp, create your manifests,
seal your secrets with [seal-secrets](https://github.com/bitnami-labs/sealed-secrets) then add your
sealed secrets to your gitops repository.Seal your secrets in one command line:kubeseal <secrets.yaml -o yaml
Package: gcpsecurity
This Package helps to improve GCP project security and reduce billing from unused resources by performing some checksThis script is worked for individual GCP project not on organization or folder levelpackage installationpip3 install gcpsecurityEnsure google cloud platform API is enable for particular service on which you want to perform checkexample,

Enable api for data proc
Enable api for app engine ...etcHow to use ??step 1: Import classes to perform checks

    from gcpsecurity.gcp_data_proc import ExecuteCheckDp
    from gcpsecurity.gcp_gke import ExecuteCheckGke

step 2: Create GCP service account with project viewer permission
step 3: Initlize ExecuteCheck class with service account file path and project id

    dp = ExecuteCheckDp(servive_account_file_path=SERVICE_ACCOUNT_FILE_PATH, project_id=PROJECT_ID)
    gke = ExecuteCheckGke(servive_account_file_path=SERVICE_ACCOUNT_FILE_PATH, project_id=PROJECT_ID)


step 4: Call perform_check() method of ExecuteCheck classes object

        dp_result = dp.perform_check()
        gke_result = gke.perform_check()

step 5: Print results

        print(dp_result)
        print(gke_result)Available services checks classes:compute engine checks ==> from gcpsecurity.gcp_vm import ExecuteCheckVm
    IAM checks            ==> from gcpsecurity.gcp_iam import ExecuteCheckIam
    VPC checks            ==> from gcpsecurity.gcp_vpc import ExecuteCheckVpc
    cloud storage         ==> from gcpsecurity.gcp_gcs import ExecuteCheckGcs
    cloud sql             ==> from gcpsecurity.gcp_cloud_sql import ExecuteCheckSql
    app engine            ==> from gcpsecurity.gcp_app_engine import ExecuteCheckGae
    data proc             ==> from gcpsecurity.gcp_data_proc import ExecuteCheckDp
    kubernetes engine     ==> gcpsecurity.gcp_gke import ExecuteCheckGkeExample scripthttps://github.com/Aadeshkale/gcp-security/blob/master/main.py** Notes :-Script might take time to execute because it is make googleapis calls.Service account file should have appropriate permissions to perform checks That is Project Viewer (You can set permissions as per service also)To add other checks as per your use go to package gcpsecurity add checks in existing services scripts or add new scripts for new services
reference :-https://github.com/Aadeshkale/gcp-security/tree/master/gcpsecurity
Package: gcp-simple-pipeline
GCP simple pipeline package is implemented for deploying a Continuous Deployment or Delivery system (CD) by GCP services.You can use this simple pipeline for deploying your personal solution in 2 environments: staging and production.It is part of theeducational repositoriesto learn how to write stardard code and common uses of the TDD, CI and CD.PrerequisitesComing soon.InstallationComing soon.Change LogSeeCHANGELOG.mdfor details.LicenseThis package is released under the MIT license.  SeeLICENSEfor details.
Package: gcp-taskqueue
fastapi-authenticatorfastapi authenticator for google cloud tasksInstallationserver:pip3 install fastapi-authenticatorclient:pip3 install gcp-taskqueueUsageserver:fromfastapiimportDepends,FastAPIfromfastapi_authenticatorimportGoogleCloudTask,google_cloud_task,google_cloud_authapp=FastAPI()@app.post("/task1")deftask_handler(claims:dict=Depends(google_cloud_auth),task:GoogleCloudTask:Depends(google_cloud_task)):...client:fromgcp_taskqueueimportTaskQueuequeue=TaskQueue(queue_id="your-queue-name")queue.create_http_task("https://url",deadline=300)DeploymentService Account UserRole is needed for the client to generate oidc token.
Package: gcpTokenGenerator
how to usefromgcpTokenGeneratorimporttokenGeneratorconf={"STG_Conf":{"account":'{your service account}',# need to get from dev ops"host":'{your project id}',"target_audience":'{your client id}',"iap":True,},"PRD_Conf":{"account":'{your service account}',# need to get from dev ops"host":'{your project id}',"target_audience":'{your client id}',"iap":True,}}instance=tokenGenerator(conf)token=instance.generateToken('STG_Conf')print('token',token)when reusing same instance, generated token will renew the token only if it's expired after one hourmeaning it remember the last token generated to one of the configs was used on initexample:fromgcpTokenGeneratorimporttokenGeneratorconf={"STG_Conf":{"account":'{your service account}',# need to get from dev ops"host":'{your project id}',"target_audience":'{your client id}',"iap":True,},"PRD_Conf":{"account":'{your service account}',# need to get from dev ops"host":'{your project id}',"target_audience":'{your client id}',"iap":True,}}instance=tokenGenerator(conf)stg_token=instance.generateToken('STG_Conf')prd_token=instance.generateToken('PRD_Conf')print('stg_token',stg_token)print('prd_token',prd_token)# this will generate new token only if it's expiredstg_token=instance.generateToken('STG_Conf')prd_token=instance.generateToken('PRD_Conf')
Package: gcptool
Carve Systems GCP Assessment ToolingInstallationThis tool requires Python 3.8 or newer.Additionally, thegcloudCLIis required for authentication against Google user accounts.UsagePrerequisitesTo use this tool against a Google Cloud environment, you must be have a user account with theSecurity Reviewerrole granted across all relevant projects. The use of a service account with these permissions is recommended if possible; regular user accounts are subject to stricter rate-limiting by Google.Currently, gcptool authenticates using Google Application Default Credentials. If you are using a regular user account for testing, thegcloud auth application-default logincommand will properly configure credentials for you. In the case of a service account, setting theGOOGLE_APPLICATION_CREDENTIALSenvironment variable to a path containing a service account keypair will be required.You can verify access is working correctly using two commands within the tool;gcptool list-projectswill print a list of projects you can currently access, andgcptool check <PROJECT-NAME>will verify your account has the necessary set of permissions to perform all scans.Running TestsOnce you've verified access, thegcptool scancommand can be used to perform testing. To run a basic scan, the command can be used as follows:gcptool scan PROJECT-NAME,PROJECT-NAME-2 path/to/report/findings/folderThis will produce several outputs:A JSON inventory of Google Cloud Resources ingcptool_cache.jsonin the current directory (NOTE: this file should be handled with care, as it may contain sensitive information)CDPS-format Markdown findings ready for editing and use in a report (Note that some of these are guidance to possible points of interest for manual testing)gcptool.loglog file with more detailed informationgcptool will use the generated cache file when possible to avoid making extraneous API requests; the--cache-onlyoption can be used to exit with an error if any scan would required fresh data from the API. This is convenient, for example, to run new scans against already-collected data.Other useful arguments are:--service: only run scans for a specific GCP service. For example,--service computewill only run Compute Engine tests.--scan: only run a specific scan. One particularly useful use for this argument is--scan inventory, which will gather data or a complete cache file but will not run any analysis. (gcptool list-scansprovides a helpful list of scans and services)DevelopmentDevelopment dependenciesCode formatting is handled using thepre-committool. Please install and configure it before making any commits!
Package: gcptree
gcptreeLike the unixtreecommand but for GCP Org Heirarchy.For a note on coloring, the org node is green, folders and blue, and projects that are notACTIVEare dimmed. These styles won't show up if you redirect output and Windows is supported by virtue of thecoloramapackage.Installationpip install gcptreeUsageFirst make sure you are authenticated togcloudand have your application default credentials set using:gcloud auth login --update-adcYou just need your org id, which you can find in the GCP console under the popup for picking organizations. It should be a long number around 12 digits or so.gcptree $ORG_IDThe flags are fairly simple and can be seen with the--helpflag.$ gcptree --help
usage: gcptree [-h] [--format FORMAT] [--full-resource] org_id

Print out a GCP org heirarchy

positional arguments:
  org_id           GCP Organization ID

optional arguments:
  -h, --help       show this help message and exit
  --format FORMAT  Output format (json or text)
  --full-resource  API-parsable nodes where org and folder resource names are not resolved, i.e org/123 instead of example.comPermissionsThe minimum role you'll needat the organization levelis:roles/browserroles/cloudasset.viewer
Package: gcpts
GCP time seriesRequirementsPython 3.10+Installationpip install gcptsTestpoetry run pytest -s -vvUsageimportgcptsimportpandasaspdimportnumpyasnpgcpts_client=gcpts.GCPTS(project_id="example_project",dataset_id="example_dataset")# Prepare example data, your data need to have 3 columns named symbol, dt, partition_dtdf=pd.DataFrame(np.random.randn(5000,4))df.columns=['open','high','low','close']# symbol represent a group of data for given data columnsdf['symbol']='BTCUSDT'# timestamp should be UTC timezone but without tz infodf['dt']=pd.date_range('2022-01-01','2022-05-01',freq='15Min')[:5000]# partition_dt must be date, data will be updated partition by partition with use of this column.# Every time, you have to upload all the data for a given partition_dt, otherwise older will be gone.df['partition_dt']=df['dt'].dt.date.map(lambdax:x.replace(day=1))gcpts_client.upload(table_name='example_table',df=df)# Query for raw data.raw_clsoe=gcpts_client.query(table_name='example_table',field='close',start_dt='2022-02-01 00:00:00',# yyyy-mm-dd HH:MM:SS, inclusiveend_dt='2022-02-05 23:59:59',# yyyy-mm-dd HH:MM:SS, inclusivesymbols=['BTCUSDT'],)# Query for raw data with resamplingresampeld_daily_close=gcpts_client.resample_query(table_name='example_table',field='close',start_dt='2022-01-01 00:00:00',# yyyy-mm-dd HH:MM:SS, inclusiveend_dt='2022-01-31 23:59:59',# yyyy-mm-dd HH:MM:SS, inclusivesymbols=['BTCUSDT'],interval='day',# month | week | day | hour | {1,2,3,4,6,8,12}hour | minute | {5,15,30}minuteop='last',# last | first | min | max | sum)DisclaimerThis allows you to have SQL injection. Please use it for your own purpose only and do not allow putting arbitrary requests to this library.
Package: gcputil
GCP Utils PackageIncludes the following Methods:def  upload_to_gcp_storage(bucket, src_filename, dest_filename, folder='')Function to upload a file to google cloud storage
 :parameter bucket: name of bucket to upload files to
 :parameter src_filename: name of source filename to upload
 :parameter dest_filename: specify name of file once uploaded
 :parameter folder: optional folder path to upload to. (e.g 'folder1/')def  download_from_gcp_storage(bucket, src_filename, dest_filename, folder=''):Function to download a file to google cloud storage
 :parameter bucket: name of bucket to download files from
 :parameter src_filename: name of source filename to download
 :parameter dest_filename: specify name of file once downloaded
 :parameter folder: optional folder path to download from. (e.g 'folder1/')def  list_gcp_bucket_blobs(bucket, prefixStr=''):Function to list bucket from google cloud storage
 :parameter bucket: name of bucket to grab items from
 :parameter prefixStr: optional prefix for files to list in bucket
 :return array: returns array of bucket items in blob format
Package: gcp-utility
GCP Utility toolWe can backfill any kind of bigquery sql to specific destination table with powerful python jinja templating feature. By default it can support date templates in your query. It will automatically convert to date template to date value in your query while executing.Following are the available date templates you can use it in your query.{{ds}}-YYYY-DD-MMdatestring{{ds_nodash}}-YYYYDDMMdatestring{{ds_yesterday}}-YYYY-DD-MMyesterdaydatestring{{ds_yesterday_nodash}}-YYYYDDMMyesterdaydatestring{{ds_tomorrow}}-YYYY-DD-MMtomorrowdatestring{{ds_tomorrow_nodash}}-YYYYDDMMtomorrowdatestringGetting StartedInstallingUse virtualenv to Install the dependancy packagespip install virtualenv
virtualenv venv
. ./venv/bin/activate##### Windows:
python -m pip install virtualenv
python -m virtualenv venv
./venv/Scripts/activateInstall bq_client command line tool using pippipinstallgcp-utility-h command will return the help content which contains all the available  optionsbq_client -hEnd with an example of getting some data out of the system or using it for a little demoAvailable optionusage:bq_client.py[-h][-qQUERY][-f][-pPROJECT][-ssql][-dml][-dDESTINATION_TABLE][-wWRITE_DESPOSITION][-qfQUERY_FILE][-tTEMPLATE][-tfTEMPLATE_FILE][-sSTART_DATE][-eEND_DATE][-sfSERVICE_ACCOUNT_PATH]optionalarguments:-h,--helpshowthishelpmessageandexit-qQUERY,--queryQUERYprovidevalidbigquerysql-f,--no-flatternFlatternresults-pPROJECT,--project_idPROJECTprovidevalidprojectid-ssql,--standard-sqlMentionifusingStandardsql-dml,--dml-statementMentionifusingDMLstatementsinyourquery-dDESTINATION_TABLE,--destination-tableDESTINATION_TABLE<projectname>:<datasetid>.<tableid>providevaliddestinationproject-id-wWRITE_DESPOSITION,--write-despositionWRITE_DESPOSITIONWritedispositionvalue-qfQUERY_FILE,--query-fileQUERY_FILEprovidebigquerysqlfilepath-tTEMPLATE,--templateTEMPLATEprovidetemplatevalues-tfTEMPLATE_FILE,--template-fileTEMPLATE_FILEprovidetemplatefilepath-sSTART_DATE,--start-dateSTART_DATEProvidevalidstartdate(YYYY-MM-DD)-eEND_DATE,--end-dateEND_DATEProvidevalidenddate(YYYY-MM-DD)-sfSERVICE_ACCOUNT_PATH,--service-account-file-pathSERVICE_ACCOUNT_PATHprovidevalidpathofserviceaccountjsonfilepassing raw queryyou can pass the raw query to script that need to be executebq_client -q "select * from [sample_proj:datasetid.tableid]" -p destination_projectid -d dest_datasetid.tableidUsing sql fileif your query is quite long, you can save it to text file and pass the path of itbq_client --query-file "/path/sample.sql" -p destination_projectid -d dest_datasetid.tableid```Passing date parameterIn default it will substitute current date into template fields. but if you want to change the date value you can pass manually using start date parameterbq_client --query-file sample.sql -p projectid -d dest_datasetid.tableid -s 2018-02-02BackfillPass start-date and end-date to script, it will backfill the data.bq_client --query-file sample.sql -p projectid -d dest_datasetid.tableid -s 2018-02-02 -e 2018-02-10DevelopmentWhat things you need to install the software and how to install themgoogle-cloud-bigquery==1.24.0
Jinja2==2.11.2
Package: gcrc
Google Container Registry Cleanup utilityIf you use a CI system to push images to Google Container Registry you will eventually run into a situation where you're paying more than you would like for the storage there.This tool helps you keep those costs reasonable, while allowing you to benefit from caches, keeping backups for rollbacks, etc.Prerequisites and setupPython 3.9+Gcloud SDKwith an authentication token that has admin access to the registry properly configuredpipinstall-UgcrcUsageReplacegcr.io/project-namewith the specific GCR address to your repository, e.g. if your Google Cloud project is calledfoo-barand it's in the EU zone this would likely beeu.gcr.io/foo-bar.List all imagesFigures out the different images we can access in the Container Registry.gcrclist-imagesgcr.io/project-nameShow information on imagesAnalyzes current images and any need for cleanup based on current configuration without actually deleting anything. Use to check that configuration seems correct.gcrcimage-infogcr.io/project-nameClean up imagesLook up images with unneeded tags and delete them from Google Container Registrygcrccleanupgcr.io/project-nameConfigurationThe following environment variables can be used to adjust the configuration:KEEP_TAGS_MIN=10Keeps at least this many tags for every image.KEEP_TAGS_DAYS=14Keep everything from within this many days.KEEP_EXTRA='["^latest$", "^(master|main)-"]'List of regex matches for important images that we want to keep an extraKEEP_TAGS_MINitems of. Formatted as a JSON list of strings.The default is to keeplatestand any tags starting withmaster-ormain-for people who tag their images like<branch>-<timestamp or commit hash or uuid>.You can also put these in a.env-file, but environment variables take priority over.env.# .env
KEEP_TAGS_MIN=10
KEEP_TAGS_DAYS=14
KEEP_EXTRA='["^important-", "^latest$"]'Financial supportThis project has been made possible thanks toCocreatorsandLietu. You can help us continue our open source work by supporting us onBuy me a coffee.
Package: gcreate
A command-line utility to bootstrap GitHub repositories with labels and
webhooks.InstallationTo get the latest stable release from PyPipipinstallgcreateTo get the latest commit from GitHubpipinstall-egit+git://github.com/bdero/gcreate.git#egg=gcreateGenerate an initial configuration in$HOME/.gcreateand modify itgcreateinitvi~/.gcreate# Configuration directions are located insideRelease NotesNo releases yet.
Package: gc-recurly
Recurly’s Python client library is an interface to itsREST API.Python VersionsThe minimum supported requirements for this library are:Python >=2.7.9OpenSSL >=1.0.1InstallationRecurly is packaged as a Python package. We recommend you install it withPyPIby adding it to yourrequirements.txt:pip install recurlyConfigurationSet your API key and optionally set a certificate authority certificate file and default currency:import recurly

recurly.SUBDOMAIN = 'your-subdomain'
recurly.API_KEY = '012345678901234567890123456789ab'

# Set a certificate authority certs file to validate Recurly's certificate
recurly.CA_CERTS_FILE = '/etc/pki/tls/certs/ca-bundle.crt'

# Set a default currency for your API requests
recurly.DEFAULT_CURRENCY = 'USD'Recurly Python Client TestsTo run these tests in Python 2.7, use theunittesttest runner:$ python -m unittest discover -s testsUnder Python 2.6 or earlier, install theunittest2distribution and use it
instead:$ pip install unittest2  # or easy_install
$ python -m unittest2 discover -s testsThe resource tests intest_resources.pywill run using the HTTP fixtures intests/fixtures. To run the tests against a live Recurly API endpoint,
configure your Recurly test account and use its API key in theRECURLY_API_KEYenvironment variable:$ RECURLY_API_KEY=1274…54e3 python -m unittest tests.test_resourcesThe live Recurly API endpoint can also be tested while validating the server
certificate with theRECURLY_CA_CERTS_FILEenvironment variable, which should
be a filename of concatenated certificate authority X.509 certificates:$ RECURLY_API_KEY=1274…54e3 RECURLY_CA_CERTS_FILE=/etc/pki/tls/certs/ca-bundle.crt -m unittest tests.test_resourcesUsagePlease see theRecurly APIfor more information.Supporthttps://support.recurly.comstackoverflowIRCIf you have general questions about the library or integration, you may find some of us in the #recurly irc channel on the Freenode network.