Saravanan666/pypi-org huggingface dataset:
Package: bq-sqoop
This project is to just have a simple cli command to export data from ES, postgres, etc using the CPU’s,
It’s intended to be used in Data workflow for extracting data out.NoteThis is still early in the development and a bit rough around the edges.
Any bug reports, feature suggestions, etc are greatly appreciated. :)Installation and usageInstallationSince this is a Python package available on PyPi you can install it like
any other Python package.# on modern systems with Python you can install with pip$pipinstallbq-sqoop# on older systems you can install using easy_install$easy_installbq-sqoopUsageThe commands should be mostly self-documenting in how they are defined,
which is made available through thehelpcommand.$bq-sqoopusage:bq-sqoop-harguments:-h,--helpshowthishelpmessageandexit-v,--versionShowversionandexit.-cCONFIG_FILE,--config_fileCONFIG_FILETomlConfigfileforthebq-sqoopjob.CanbealocalfilepathorapublichttplinkoraGCSfileeg,https://storage.googleapis.com/sample_config.tomlorgs://gcs_bucket/sample_config.tomlor/tmp/sample_config.toml-d,--debugDebugmodeon.Configuration filesYou can find an example repository athttps://github.com/therako/bqsqoop-examples.gitConfiguration objectsBigqueryExtractorElasticsearchSQLBigquery[bigquery]project_id="destination-google-project-id"dataset_name="destination-dataset"table_name="destination-table-name"gcs_tmp_path="gs://gcs-tmp-bucket/bqsqoop/"ExtractorElasticsearch[extractor.elasticsearch]url="localhost:9200,localhost:9201"index="source-es-index-name"timeout="60s"scroll_size=500fields=["_all"]SQL[extractor.sql]sql_bind="postgresql+psycopg2://username:password@127.0.0.1:5432/database"query="select * from table_name"
Package: bqt
BQtAdd QT support to Blender, letting you create custom UI for your addons with PySide2 QtWidgets.Featuresfeaturedescriptionfully custom UIInstead of feeling limited by N-Panel only UI. Do whatever you want.cross appQt widgets run nativaly in Krita, 3ds Max, Maya, … and are great to use in cross dcc pipelinesexisting communityhundreds of QT widgets on GitHub you can reuse, and lots of stack exchange questions answered regarding qt.ThemedBQt applies theblender-qt-stylesheetso qt widgets will by default look similar to Blender.RequirementsBlender2.83or higher.Windows or MacPySide2PyObjC&iterm2(Mac only)How to useInstall & enable the bqt addon (seeinstallation docs)Create & show your Qt widget, bqt will automatically register it with it's widget manager.Thebqt env varslet you toggle bqt features on and offsample code:bqt_demoshows you how to use bqt with several qt demos you can run in Blendercontributei'm a technical writer and want to contribute to the docs. seereadmeon the wiki.i'm a dev and want to contribute to bqt code, seecontribute guidelines.i'm a user or tester and want to report a bug or request a feature, please posthereCommunityDiscuss BQt onthe BlenderArtiststhreadYnput  thread
Package: bqtools
Python Tools for BigQueryWhy?For data collection and data exploration, we like to work withBigQuery. But we have not found a python library, to easily handle recurring tasks like adding new data (of potentially inconsistent schema) and schema migrations. So we took a couple of our solutions for those tasks and put them into this library.What?bqtoolsprovides a light-weight solution to explicit schema management with python-native types (unlike pandas dtype) and
some convenient type checking, inference and conversions. Table-objects created bybqtoolscan be read from BigQuery, stored locally, read from a local file and written to BigQuery. Table schemas can be changed and data can be added or modified.Installpipinstall--upgradebqtoolsExamples:Create basic tablesfromfourtytwoimportbqtoolsschema=[{'name':'number','field_type':'INTEGER'},{'name':'text','field_type':'STRING'},{'name':'struct','field_type':'RECORD','mode':'REPEATED','fields':[{'name':'integer','field_type':'INTEGER'},{'name':'text','field_type':'STRING'}]}]# valid BigQuery types see:# https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types# geo and array are currently not/not fully supported# data = columns of liststable=bqtools.BQTable(schema=schema,data=[[1,2,3,4],['a','b','c','d']])# data = rows of dictstable=bqtools.BQTable(schema=schema,data=[{'number':1,'text':'a'},{'number':2,'text':'b'},...])View dataprint(table.data)# list of all columnsprint(table.rows(n=10))# list of first n rows# convert to pandas.DataFramedf=table.to_df()# warning: pandas dtypes may be inconsistent# with BigQuery Schema field_typesAppend datarows=[{'number':5,'text':'e'}]table.append(rows)row=[[6,'f']]table.append(rows)Load table from BigQuery# requires environment variable GOOGLE_APPLICATION_CREDENTIALS# or parameter credentials='path-to-credentials.json'table=bqtools.read_bq(table_ref='project_id.dataset_id.new_table_id',limit=10,# limit query rowsschema_only=False# set True to only add data)Modify table schema# change column order and field_typenew_schema=[{'name':'text','field_type':'STRING'},{'name':'number','field_type':'FLOAT'},]table.schema(new_schema)# change column namestable.rename(columns={'number':'decimal'})Write table to BigQuery# requires environment variable GOOGLE_APPLICATION_CREDENTIALS# or parameter credentials='path-to-credentials.json'table.to_bq(table_ref,mode='append')Persist tables locally# write to local file (compressed binary format)table.save('local_table.bqt')# load from local filetable=bqtools.load('local_table.bqt')
Package: bqtools-json
This is bqtools a handy set of utils to help create big query tables from json objects and make them into tables (so get the schema) and to create json new line files.
Plus a set of common functions to ease writing biq query python codeGoalsSimplify handling and move between big query stryuctured data and json dataAllow you to create big query table schemas from json structures (as resource or schema objects) uses reflection of typesProvides easy generation of views for day partitioned tabledhead - latest datadiff views for time snap shots of data i.e. each day partition has current view of dataCalculate valid json structures from representative json data that can be used as basis of big query schemasClean json data such that it can be loaded into big queryReplace bare lists with dictionariesReplace field  names with valid values that can be column names in big query (removes spaces, characters not allowed in field names using same algorithms big query uses when auto detecting schemas)Encodes json output of dates, datetimes, times, timedeltas encoded in format acceptable for big query corresponding field typesGenerate code for bq command line tool from bq table structuresSimplify common tasks of handling big query dataBasic tests on dataset or tables existingSchema patching compare an existing schema to a template json object calculate if changed and generate a merged schema that can be used in a patchFlattening views to avoid view depth limitsimport bqtools

# if you load a json object say something like
foo = {
        "id":1,
        "description":""
        "aboolean":False
      }
      
# generate a schema
table = {
   "type":"TABLE",
   "location":os.environ["location"],
   "tableReference":{
       "projectId": os.environ["projectid"],
       "datasetId": os.environ["dataset"],
       "tableId": key
   },
   "timePartitioning":{
       "type": "DAY",
       "expirationMs": "94608000000"
   },
   "schema": {}
}

# use bqtools to create a schema structure
table["schema"]["fields"] = bqtools.get_bq_schema_from_json_repr(foo)Demonstrates some of power of tools viabqsyncthat is installed if you install via pip.pip install bqtools-jsonOr you can find the source for thishereAlso provides means to handle views across environments supports SQL views based upon Jinja templates. So you can "configure" views for different environments.Provides support for calculating view dependencies such views can be applied in parallel tranches.Provides support for calculating authorised views based on set of views "compiled"Compilation flattens views to lower view depth but automatically does not flatten authorised views to keep access to data tight as feasible.
Package: bqtxflowutils-2
beepbeep_txflowutilsThis is an project to Sync data transformation information from BigQuery to GitHub and create a dynamic Markdown documantation.InstallationRun the following to install:pipinstallbeepbeep_txflowutilsUsagegitclonehttps://github.com/beepbeeptechnology/sync_from_bigquery_to_github.gitCreate venvpython-mvenv<venv_name>source<venv_name>/Scripts/activateforwondowsosGenarate streamlit appstreamlit run app.pyDeveloping beepbeep_txflowutilsTo install beepbeep_txflowutils, along with the tool need to develop and run test, run the following in your virtualenv:$pipinstall-e.[dev]TODO
Package: bquest
BQuestEffortlessly validate and test your Google BigQuery queries with the power of pandas DataFrames in Python.We would like to thankMike Czechwho is the original inventor of bquest!WarningThis library is a work in progress!Breaking changes should be expected until a 1.0 release, so version pinning is recommended.OverviewUse BQuest in combination with your favorite testing framework (e.g. pytest).Create temporary test tables fromJSONorpandas DataFrame.Run BQ configurations and plain SQL queries on your test tables and check the result.InstallationVia PyPi (standard):pipinstallbquestVia Github (most recent):pipinstallgit+https://github.com/ottogroup/bquestBQuest also requires a dedicated BigQuery dataset for storing test tables, e.g.resource "google_bigquery_dataset" "bquest" {dataset_id    = "bquest"friendly_name = "bquest"description   = "Source tables for bquest tests"location      = "EU"default_table_expiration_ms = 3600000}We recommend setting anexpiration timefor tables in the bquest dataset to assure removal of those test tables upon
test execution.ExampleGiven a pandas DataFramefooweightprediction_datebar2320190301my4220190301and its table definitionfrombquest.tablesimportBQTableDefinitionBuildertable_def_builder=BQTableDefinitionBuilder(GOOGLE_PROJECT_ID,dataset="bquest",location="EU")table_definition=table_def_builder.from_df("abc.feed_latest",df)you can use the config file./abc/config.py{"query":"""
        SELECT
            foo,
            PARSE_DATE('%Y%m%d', prediction_date)
        FROM
            `{source_table}`
        WHERE
            weight > {THRESHOLD}
    ""","start_date":"prediction_date","end_date":"prediction_date","source_tables":{"source_table":"abc.feed_latest"},"feature_table_name":"abc.myid",}and the runnerfrombquest.runnerimportBQConfigFileRunner,BQConfigRunnerrunner=BQConfigFileRunner(BQConfigRunner(bq_client,bq_executor_func),"config/bq_config",)result_df=runner.run_config("20190301","20190308",[table_definition],"abc/config.py",templating_vars={"THRESHOLD":"30"},)to assert the result tableassertresult_df.shape==(1,2)assertresult_df.iloc[0]["foo"]=="my"TestingFor the actual testing bquest relies on an accessible BigQuery project which can be configured
with thegcloudclient. The correspondingGOOGLE_PROJECT_IDis extracted from this project
and used withpandas-gbqto write temporary tables to the bquest dataset that has to be pre-
configured before testing on that project.For Github CI we have configured an identity provider in our testing project which allows
only core members of this repository to access the testing projects’ resources.Important LinksFull documentation:https://ottogroup.github.io/bquest/
Package: bqunit
Testing framework for BigQuery SQL.What is this?BigQuery enables us to execute “super-fast SQL queries
using the processing power of Google’s infrastructure”.However, testing query-based data pipelines sometimes become depressing work, because:SQL itself takes more responsibility in data transformation logic,
and the glue code layer like Python scripts(which is relatively easy to test) doesn’t.We can’t imitate BigQuery infrastructure easily.
There’s no Docker image, StandardSQL has many unique syntaxes which can’t be used on other RDBMS,
and above all, Google has huge computing resources than ours.BQUnit solves this problem, by managing your test data preparation
on your BigQuery data set, which is isolated from your production environment.UsageFirst, instantiate BQUnit object:bqunit = BQUnit(project_id='test-env-123456', dataset_name='bqunit')

# If Application Default Credential is set, project id is not required.
bqunit = BQUnit(dataset_name='bqunit')And then, mockup your tables by a fixture() method call:bqunit.fixture(
    table_name='your-production-123456.foo.bar',
    statement="""
    select 1 as col1, 'str_1' as col2, true as col3
    union all
    select 2, 'str_2', false
    """)You don’t need to know where to insert your test data.
You just need to specify your production table name here.Testing will be like this:tested_query = """
   select col1, col2
   from `your-production-123456.foo.bar` foo
   where col1 = 1
 """
query_result = bqunit.test_query(tested_query)
assert query_result.total_rows == 1BQUnit execute your query on test data set, which is created when you called the fixture() method,
so you can predict its result set correctly, and make assertions.Note thatquery_resultwill be RowIterator object ofgoogle-cloud-bigquerylibrary.
see alsogoogle-cloud-bigquery documentation.
Package: bqup
bqupbqup is a backup tool for BigQuery projects. It can export a BigQuery
project's structure and source code while mimicking the hierarchy of
datasets and tables.For the full story of why we made bqup, check out ourblog
post!Installationbqup can be installed usingpip.pip install bqupAlternatively, you can also clone the repository then runinstall.git clone https://github.com/thinkingmachines/bqup.git
cd bqup
python3 setup.py installUsageCommand line optionsYou can list down the options by runningbqup --help.bqup [-p PROJECT_ID] [-d TARGET_DIR] [-fvxr] [-e REGEX]

Options:
  -p PROJECT_ID, --project PROJECT_ID  Project ID to load. If unspecified,
                                       defaults to current project in
                                       configuration.
  -d TARGET_DIR, --dir TARGET_DIR      The target directory where the project
                                       will be written. Defaults to current
                                       timestamp.
  -f --force                           Overwrite target directory if it exists.
  -v --verbose                         Print a summary of the loaded project.
  -x --schema                          Export table schemata as json.
  -r --routine                         Include routines in export.
  -e REGEX, --regex REGEX              Regex pattern to filter datasets to be exported.DevelopmentSet upgcloudto run with your personal account (aka run with scissors).Set upapplication-default.gcloud auth application-default loginInstallwheel.pip3 install wheelInstall bqup.pip3 install -e .Alternatively, you can also install it using:python3 setup.py developRun bqup (seeUsage).Setting up regular backupsCheck outbqup-starterto set up regular bqups using GitHub's workflows!DistributionRunmake testto try a test upload.Runmake distto upload a distribution.Both of these will callmake build, which rebuilds the package locally.ContributingIf you wish to contribute, check out ourcontributing
guide!A list is maintained for all external contributors who have submitted pull
requests that were subsequently approved. Users are allowed and encouraged to
fork the project and submit pull requests and issues. We only request that
contributions adhere to these guidelines:Commit messagesGithub flowThe official maintainers in charge of responding to issues and merging pull
requests are:Pepe BawaganMark Steve SamsonCarlson ChengContributorsThanks to all these wonderful people who've helped out with bqup:JessRamPepe BerbaTim PronEnzoArdiebhuesemannbzieba-spartezprofwackourthoajkruseDisclaimersbqup is maintained on abest effortbasis:No amount of official time is currently being dedicated to the regular
maintenance of this project.Thinking Machines does not make any guarantees about the quality of the
software.Thinking Machines reserves the rights to:refuse to resolve issuesclose issues without resolutionrequest changes to pull requestsreject pull requests outright
Package: bq-utils
bq-utils - A collection of BigQuery utilitiesCurrently the utilities covered in this library are limited to managing table and view descriptions.
It's quite useful to have the field descriptions available on
the schema tab and on the query editor pane on the BigQuery UI and this library helps you with uploading them and copying them around.Installpipinstallbq-utilsPythonCopy descriptions between tables and viewsfromgoogle.cloudimportbigqueryfrombqutils.bigquery_description_managerimportBigQueryDescriptionManagerbq_client=bigquery.Client()description_manager=BigQueryDescriptionManager(bq_client)source_table_id='s_project.s_dataset.s_table'target_table_id='t_project.t_dataset.t_table'description_manager.copy_field_descriptions(source_table_id,target_table_id)Upload descriptions from csvfromgoogle.cloudimportbigqueryfrombqutils.bigquery_description_managerimportBigQueryDescriptionManagerbq_client=bigquery.Client()description_manager=BigQueryDescriptionManager(bq_client)descriptions_csv_path='table_descriptions.csv'target_table_id='t_project.t_dataset.t_table'description_manager.upload_field_descriptions(descriptions_csv_path,target_table_id)Command lineUsageusage:__main__.py[-h][--sourceSOURCE]--targetTARGET[--csv_pathCSV_PATH][--debug]{desccopy,descupload}CopyoruploadfielddescriptionsforBigQuerytables/views

positionalarguments:{desccopy,descupload}optionalarguments:-h,--helpshowthishelpmessageandexit--sourceSOURCEfully-qualifiedsourcetableID--targetTARGETfully-qualifiedtargettableID--csv_pathCSV_PATHpathforthecsvfile--debugsetdebugmodeon,defaultisfalseCopy descriptions between tables and viewspython-mbqutilsdesccopy--sources_project.s_dataset.s_table--targett_project.t_dataset.t_tableUpload descriptions from csvpython-mbqutilsdescupload--csv_pathtable_descriptions.csv--targett_project.t_dataset.t_table
Package: bq-validator
bq-validatorThis is a yet another python-based BigQuery query validator.Thebq query --dry_runcommand enables us to validate queries.
However, thebqcommand doesn't support service account impersonation, even though it supports workload identity federation credentials at Google Cloud SDK 390.0.0.
Thebq-validatorcommand would be useful, when we take advantage of service account impersonation to validate BigQuery queries.InstallThe package is available onpypipipinstall-Ubq-validatorHow to use$bq-validator--help

Usage:bq-validator[OPTIONS]PATHValidateBigQueryqueriesPATHiseitherofaSQLfilepathoradirectory.Whenitisadirectory,thecommandrecursivelyvalidatesallSQLfilesinthedirectory.

Options:--quota_projectTEXTBigQueryclientprojectID--client_projectTEXTBigQueryclientprojectID--client_locationTEXTBigQueryclientlocation--impersonate_service_accountTEXTImpersonateserviceaccountemail--versionShowtheversionandexit.--helpShowthismessageandexit.
Package: bqx
Generate sophisticated query for Google BigQuery in simple way.ARCHIVEDThis project is archived and no longer maintained.What is BQX?BQX is a minimal query generator for Google BigQuery. It’s mainly
intended for being used by data analysts / scientists who want to
analyze big data.BQX fires its power especially on making LONG and NESTED query. BigQuery
doesn’t accept two or more queries at once so user has to make views or
make terribly nested query. BQX’s features help you make long query
preserving high readability. See example for its features and usage.RequirementsPython 3.4 ~Installingpip install bqxExample>>> from bqx.query import Query as Q
>>> from bqx.parts import Table as T, Column as C
>>> from bqx.func import SUM
>>>
>>> shakespeare = T('publicdata:samples.shakespeare')
>>> word = C('word')
>>> q = Q().SELECT(word).FROM(shakespeare).LIMIT(10)
>>> print(q.getq())
SELECT word
FROM publicdata:samples.shakespeare
LIMIT 10See example directory for more detailed examples.LicenseThis library is BSD-licensed.Q&AHow can I use it?Running example on REPL is good for getting started, and we suggest you
to use BQX with front-ends like pandas for next step.Is it only for BigQuery? How about MySQL or else?It aims to generate query excecuted on BigQuery but it might be applied
to other SQL environments because difference between plain SQL and
BigQuery is small.We are using some compute engines which can process SQL (and dialects)
like BigQuery, Hadoop and Spark. So in near future, adding other SQL
dialect is planned.Why some functions are UPPERCASE?The first reason is for avoiding collision with Python’s reserved words.
The another reason is for SQL’s UPPERCASE manners.Is it ready to use?We hope it’s so. All main clauses are implemented from v0.2.0, but not
fully tested by hand yet.Where are documentations?We’re sorry but documentation is not available right now. It will be
published on ReadTheDocs.org soon. Stay tuned.
Package: br
NAME----__br__ (Batch Rename)SYNOPIS-------br [ __-h__ ] [ __-v__ ] [ __-o__ ] [ __-t__ ] [ __-i INDEXES__ ] [ __-d DIRECTORY__ ] [ __-p PATTERN__ ] ...DESCRIPTION-----------__BR__  (Batch Rename) is an unix utility to ease renaming of a batch of files.__-o__, __--output__cause __br__ to be verbose, showing files after they are renamed__-t__, __--test__test run, enables __--output__ too__-i__, __--indexes__ = _INDEXES_substring of indexes that need to be replaced with the new title, in form of x:y__-d__, __--directory__ = _DIRECTORY_directory containing batch of files__-p__, __--pattern__ = _PATTERN_regular expression for renaming pattern_TITLE_ new title to replace the old one__-h__, __--help__Display help information and exit.__-v__, __--version__Display version information and exit.FILES-----+ `setup.py`  		traditional easy_install or pip+ `setup.cfg` 		for distutils2 installation+ `scripts/br`		executable binary+ `br/__init__.py` 	a separate library packaged versionEXAMPLES--------```bashMrmacHD:~$ ls *.txt111111.txt 222222.txt 333333.txtMrmacHD:~$ br -t -i 2:3 -p \*.txt test111111.txt -> 11test111.txt222222.txt -> 22test222.txt333333.txt -> 33test333.txt```STANDARDS---------SEE ALSO--------AUTHOR------MrmacHD <mrmachd@gmail.com>BUGS----
Package: br4nch
br4nchis a Data Structure Tree Builder for Python. It is built in 100%/Pure Python, which means it does not use 3rd party libraries. Welistento our users inissues, Discordchanneland all over the Internetto create afast,flexibleandfriendlyData Structure Tree Builder.👀 ExampleIf you want to see more code examples and guide, please visit ourdocumentation.>>>br4nch.create.Tree(tree="Ukraine",header="Ukraine <3")>>>br4nch.create.Node(tree="Ukraine",node=["Cities","Information"])>>>br4nch.create.Node(tree="Ukraine",node=["Donetsk Oblast","Kiev Oblast"],parent="Cities")>>>br4nch.create.Node(tree="Ukraine",node="Mariupol",parent="Donetsk")>>>br4nch.create.Node(tree="Ukraine",node=["Bucha","Tsjernobyl"],parent="Kiev")>>>br4nch.create.Node(tree="Ukraine",node=["President","Area","Population"],parent="Information")>>>br4nch.create.Node(tree="Ukraine",node="Zelensky",parent="President")>>>br4nch.create.Node(tree="Ukraine",node="603.628 km2",parent="Area")>>>br4nch.create.Node(tree="Ukraine",node="44 Million",parent="Population")>>>br4nch.display.Tree(tree="Ukraine")Ukraine<3┣━Cities┃┣━DonetskOblast┃┃┗━Mariupol┃┗━KievOblast┃┣━Bucha┃┗━Tsjernobyl┗━Information┣━President┃┗━Zelensky┣━Area┃┗━603.628km2┗━Population┗━44Million⚙️ InstallationInstallbr4nchwith thepip installcommand:pip install br4nch🎯 FeaturesCreateDuplicateMoveReplaceDeleteSetResetLoadExportGetDisplay👍 ContributeIf you want to saythank youand/or support the active development ofbr4nch:Add aGitHub Starto the project.Tweet about the projecton your Twitter.Support the project by donating acup of coffee.☕ Supportbr4nchis an open source project that runs on donations to pay the bills e.g. our domain name. If you want to supportbr4nch, you can ☕buy a coffee here.‍💻 Code ContributorsHelp in testing, development, documentation and other tasks is highly appreciated and useful to the project. There are tasks for contributors of all experience levels.To get started with developingbr4nch, seeCONTRIBUTING.⚠️ Licensebr4nchis free and open-source software licensed under theGPL-3.0 License.