Saravanan666/pypi-org huggingface dataset:
Package: bigninja
bigninjaPySpark helpers to maximise data engineer productivity. Followpain-driven developmenttechnique.SetupAfterpip install bigninjastart using it byfrombigninjaimport*BigNinja works by adding extension methods to Spark's DataFrame class. All the methods start withbn_prefix to avoid conflicts with built-in methods.DataFrame.bn_select(*pattern: str), .bn_drop(*pattern: str)Select/drop columns using a wildcard pattern i.e.df.wc_select("co*")returns columns starting withco. For instance:bn_select("ci*")will select columns starting withcity.bn_select("id*", "ci*")with select both columns starting withidandciand so on..bn_display()Is like.show()buttruncateis set toFalseand arrays and structs are transformed to JSON so that you can read it..bn_union(df: DataFrame)Unions DataFrames, even if number of columns, their names and types don't match, by creating an overlap of columns from both datasets and filling missing values with null.EtcInspired by [quinn](MrPowers/quinn: pyspark methods to enhance developer productivity 📣 👯 🎉 (github.com)). Most ideas are initially taken from there.
Package: bignmf
BigNmfBigNmf (Big Data NMF) is a python 3 package for conducting analysis using NMF algorithms.NMF IntroductionNMF(Non-negative matrix factorization) factorizes a non-negative input matrix into non-negative factors. The algorithm has an inherent clustering property and has been gaining attention in various fields especially in biological data analysis.Brunet et alin theirpaperdemonstrated NMF's superior capability in clustering theleukemia datasetcompared to standard clustering algorithms like Hierarchial clustering and Self-organizeing maps.Available algorithmsThe following are the algorithms currently available. If you would like to know more about the algorithm, the links below lead to their papers of origin.Single NMFStandard Single NMFSparse NMFJoint NMFStandard Joint NMFIntegrative NMFInstallationThis package is available on the PyPi repository. Therefore you can install, by running the following.pip3installbignmfUsageThe following examples illustrate typical usage of the algorithm.1. Single NMFfrombignmf.datasets.datasetsimportDatasetsfrombignmf.models.snmf.standardimportStandardNmfDatasets.list_all()data=Datasets.read("SimulatedX1")k=3iter=100trials=50model=StandardNmf(data,k)# Runs the modelmodel.run(trials,iter,verbose=0)print(model.error)# Clusters the datamodel.cluster_data()print(model.h_cluster)#Calculates the consensus matricesmodel.calc_consensus_matrices()print(model.consensus_matrix_w)2. Joint NMFfrombignmf.models.jnmf.integrativeimportIntegrativeJnmffrombignmf.datasets.datasetsimportDatasetsDatasets.list_all()data_dict={}data_dict["sim1"]=Datasets.read("SimulatedX1")data_dict["sim2"]=Datasets.read("SimulatedX2")k=3iter=100trials=50lamb=0.1model=IntegrativeJnmf(data_dict,k,lamb)# Runs the modelmodel.run(trials,iter,verbose=0)print(model.error)# Clusters the datamodel.cluster_data()print(model.h_cluster)#Calculates the consensus matricesmodel.calc_consensus_matrices()print(model.consensus_matrix_w)Hereis the extensive documentation for more details.
Package: big-num-format
big-num-formatbig-num-format is a package which converts very large numbers into human readable text.InstallationSimply type the following in a command line to install the package:pip install big-num-formatUsageUse the format_num() function to format numbers.e.g.frombig_num_formatimport*format_num(123456789.55)# '123 million, 456 thousand and 789.55'format_num(1e50/3,shorten=True,precision=5,decimal_precision=10)# '33QDe 333qDe 333TDe 333DDe 333.3359375UDe'
Package: bigos
Bigos - do stuff on file changeUse theondecorator to run tasks when matching files are modified:from bigos import on, main

@on(r'^[^/]*/([^.][^/]*\/)*[^.][^/]*$')
def non_dotfile_task(ev):
    print 'file event:', ev.path

if __name__ == '__main__':
    main('.')Use the backend to get the raw event stream:from bigos.backend import generate_events

for event_list in generate_events():
    print event_listInstallationpip install bigosSupported platformsOn modern Linux kernels bigos will use a ctypes inotify wrapperOther systems are supported via a filesystem polling backendLicenseCopyright (C) 2014 Paweł StiasnyBigos is released under the GNU General Public License, see COPYING for
details.
Package: bigot
BigotBenchmarking library with Space and Time Complexity estimation.Pull requests are welcome !Installationpip install bigotUsageProvide a benchmark function with a single dimension parameterdefon(n):x=10000000*"-"*int(n)sleep(0.001*n)importbigotprint("Function has a space complexity of",bigot.Space(on2),"and a time complexity of",bigot.Time(on2))Function has a space complexity of O(n^2) and a time complexity of O(n^2)You can test our fancy options. See docstrings for reference.bench=bigot.Time(on2,plot=True,duration=1,verbose=True,name="My fancy function")And check the number of iterations, useful when comparing functionsprint(bench.iterations,"iterations in",bench.duration,"seconds")8 iterations in 8 secondsYou can also compare multiple functionsdefon2(n):x=10000000*"-"*int(n**2)sleep(0.001*n**2)print(bigot.Compare([on,on2]).space())Name  Duration  Iterations Space complexity
0   On       1.0        49.0             O(n)
1  On2       1.0         8.0           O(n^2)Testingpytest .
Package: bigquerydave-package
A wrapper on the bigquery libraries for simple accessGoogle install'''
pip install google
pip install google-api-python-client
pip install --upgrade google-api-python-client
pip3 install --upgrade oauth2clientpip install --upgrade google-cloud
pip install --upgrade google-cloud-bigquerygcloud auth application-default logingcloud init
'''Install with pippip install bigquerydaveimport to your python scriptfrom bigquerydave_package.bigquerydave import gcpfrom bigquerydave_package.bigquerydave import bqtry methodsgcp().list_gcp_projects()bq().list_bq_datasets()bq().list_bq_tables('watchful-lotus-364517.dave')bq().estimate_query('SELECT CURRENT_TIMESTAMP')
Package: bigquery-dry-run
OverviewThis application is used to perform a BigQuery dry run of all.sqlfiles in a folder and its subfolders.Its purpose it so ensure that all SQL files do not have any syntactic errors and make it easier for team members to quickly check a large number of files.GCP configurationAllowing access to GCP should be done following one of the methods detailed in theGCP documentation.The application will rely on your environment having been set up for authentication to GCP (e.g. throughgcloud initorenvironement variablecontaining service account credentials), it does not provide a mechanism to receive authentication credentials directly.UsageThe application will install a shell command calledbqdrywhich is simply passed a folder. It will then traverse the folder and perform a dry run of all.sqlfiles found in the folder and any sub folders. The results will be displayed on in the terminal.For example:$ bqdry my-awesome-project

>File: my-awesome-project/demo.sql
>  Result: Failed
>  Errors: None
>
>Total: 3
>Succeeded: 2
>Failed: 1bqdry -hwill provide usage information. E.g.usage: bqdry [-h] [-t THREADS] [-v] folder

Dry run of all `.sql` files in folder and subfolders.

positional arguments:
  folder                Top level folder to start scanning for `.sql` files.

optional arguments:
  -h, --help            show this help message and exit
  -t THREADS, --threads THREADS
                        Number of threads for concurrent running of queries. Defaults to 2.
  -v, --verbose         Show all file results, not just failures.
Package: bigquery-erd
bigquery-erdEntity Relationship Diagram (ERD) Generator for Google BigQuery, based uponeralchemy.ExamplesERD for a NewsMeme database schema (taken from the original project).Installationpipinstallbigquery-erderalchemyrequiresGraphVizto generate the graphs and Python. Both are available for Windows, Mac and Linux.UsageUsage from PythonFind an example notebookhere.But Wait, BigQuey is not a Relation Database?That's right. You cannot enforce primary or foreign key constraints in BigQuery. However, that doesn't mean that you should not be able to have logical dependencies between tables.Defining Relations through Column DescriptionsWe use the column description field in BigQuery to define relations between columns in a format that we can later parse programmatically.Let's assume we have a tableawith a columnidand another tableawith a columna_idthat serves as a foreign key relation toa.id.
We then add the following description tob.a_id:-> b.idDefining Relations to Datasets ExplicitlyPer default, we assume that the related tables are located inside the same dataset. However, you can also define the datasets explicitly. This is especially useful if the two related tables are not located within the same dataset.Let's assume that tableais located in datasetd1and tablebis located ind2.
The description inb.a_idwould then be:-> d1.a.idDefining Cardinality ExplicitlyCardinality defines the relationship between two tables. This package understands four different cardinalities:*, meaning "0..N"?, meaning "{0,1}"+, meaning "1..N"1, meaning "1"Per default, we assume a cardinality of*:1. You can also define the relation's cardinality explicitly.Let's assume that every record inahas at least 1 related record inb.
the description inb.a_idwould be:-> +:1 a.idExampleYou can find a example Google BigQuery project for the NewsMeme schema with annotated descriptionshere.Defining Custom Description RegExThe default RegEx for relations in column descriptions is->\s([?*+1 ]:[?*+1 ]\s)?(.*\.)?(.*)\.(.*)$. You can define a custom RegEx by setting theGBQ_RELATION_PATTERNenvironment variable.
The RegEx should match four capture groups, where:The first group is the cardinality (which is optional)The second group is the dataset id (which is optional)The third group is the table idThe fourth group is the column
Package: bigQueryExporterEnhanced
CreditThis repository is forked fromhttps://github.com/IcarusSO/bigQueryExporterChange Log (Compare with original version)query_to_local()Create a temp table with a random hash on BQ as output table, so that simultaneous execution of the function will not overwrite each other.Remove the temp table after the execution. (May also set keep_temp_table=True if you wish to keep them).bigQueryExporterExport query data from google bigquery to local machineInstallationpip install bigQueryExporterEnhancedPrepare for the connectionfrom bigQueryExport import BigQueryExporter
bigQueryExporter = BigQueryExporter(project_name, dataset_name, bucket_name)Query To TablebigQueryExporter.query_to_table(query, job_name, dataset_name)Table To GSbigQueryExporter.table_to_gs(destination_table, job_name)GS To LocalbigQueryExporter.gs_to_local(bucket, job_name, data_dir_path)Query To GS (Query to Table + Table to GS)bigQueryExporter.query_to_gs(query, job_name)Query To Local (Query to Table + Table to GS + GS to Local)export_path = bigQueryExporter.query_to_local(query)

# or with the options
export_path = bigQueryExporter.query_to_local(query, 
                                              job_name='simple_query', 
                                              data_dir_path='out/',
                                              keep_temp_table=False,
                                              overwrite_output_folder=True)RequirementYour server/ local machine should have the right to access the projectRight should be granted following the insturction onGoogle SDKExecute the following commandgcloud auth application-default login
Package: bigquery-fdw
bigquery_fdw is a BigQuery foreign data wrapper for PostgreSQL usingMulticorn.It allows to write queries in PostgreSQL SQL syntax using a foreign
table. It supports most of BigQuery’sdata
typesandoperators.Features and limitationsTable partitioning is supported.You can use partitions in your SQL
queries.Queries are parameterized when sent to BigQueryBigQuery’s standard SQL support (legacy SQL is not supported)Authentication works with a “Service
Account” Json private keyRead more.RequirementsPostgreSQL >= 9.5 up to 14Python >= 3.4Get startedUsing dockerSeegetting started with DockerInstallation on Debian/UbuntuDependencies required to install bigquery_fdw:You need to install the following dependencies:# Install required packagesaptupdateaptinstall-ypostgresql-server-dev-14python3-setuptoolspython3-devmakegccgitAll PostgreSQL versions from 9.2 to 14 should be supported.Installation# Install Multicorn
# pgsql-io/multicorn2 is a fork of Segfault-Inc/Multicorn that adds support for PostgreSQL 13/14.
# Alternatively, up to PostgreSQL 12, you can use gabfl/Multicorn that adds better support for Python3.
# You may also choose to build against the original project instead.gitclonehttps://github.com/pgsql-io/multicorn2.gitMulticorn&&cdMulticornmake&&makeinstall# Install bigquery_fdwpip3installbigquery-fdwMajor dependencies installed automatically during the installation
process:Google Cloud
BigQueryMulticornAuthenticationbigquery_fdw relies onGoogle Cloud API’s default
authentication.Your need to have an environment variableGOOGLE_APPLICATION_CREDENTIALSthat has to be accessible by
bigquery_fdw. Setting environment variables varies depending on OS but
for Ubuntu or Debian, the preferred way is to edit/etc/postgresql/[version]/main/environmentand add:GOOGLE_APPLICATION_CREDENTIALS = '/path/to/key.json'Restarting PostgreSQL is required for the environment variable to be
loaded.UsageWe recommend testing theBigQuery client
connectivitybefore trying to use the FDW.Withpsql:CREATEEXTENSIONmulticorn;CREATESERVERbigquery_srvFOREIGNDATAWRAPPERmulticornOPTIONS(wrapper'bigquery_fdw.fdw.ConstantForeignDataWrapper');CREATEFOREIGNTABLEmy_bigquery_table(column1text,column2bigint)SERVERbigquery_srvOPTIONS(fdw_dataset'my_dataset',fdw_table'my_table');OptionsList of options implemented inCREATE FOREIGN TABLEsyntax:OptionDefaultDescriptionfdw_datasetBigQuery dataset
namefdw_tableBigQuery table namefdw_convert_tzConvert BigQuery
time zone for dates
and timestamps to
selected time zone.
Example:'US/Eastern'.fdw_group'false'SeeRemote
grouping and
counting.fdw_castingSeeCasting.fdw_verbose'false'Set to'true'to output debug
information in
PostrgeSQL’s logsfdw_sql_dialect'standard'BigQuery SQL
dialect. Currently
onlystandardis supported.More documentationSeebigquery_fdw documentation.
Package: big-query-kusto-client
big-query-kusto-clientPackage to facilitate the gathering of big datasets
from Azure Data Explorer by paginatingUsagefrombigquerykustoclientimportBigQueryKustoClient# Resolve the KQL client how ever you see fitkusto=build_kusto_client()# pass it on the constructor# and much better using with:withBigQueryKustoClient(kusto)asclient:df:pandas.DataFrame=client.execute_query(db='ContosoSales',query='SaltesTable | order by DateKey, ProductKey, CustomerKey',optimal_page=True)len(df)# Will give you the amount of rows it broght for youImportant NoteFor this to work it is required that the query imposes an
order on the results, no matter what column you use but an| order byoperator must be in the query.Other considerationsThe package uses this other values from the system for its
configuration. It tries to keep sensible defaults:ADX_RECORDS_LIMIT=500000  # Limit of amount of rows in ADXADX_SIZE_IN_BYTES_LIMIT=67108864  # 64MB limit of size of resultBQKC_PAGE_SIZE=100000  # Default pagesize that we'll use to paginate the resultsBQKC_SQ_PREFIX=BigQueryKustoClient  # A prefix that will be used as namespace for the queriesBQKC_PAGE_SIZEwill get overridden if the parameteroptimal_page=Trueis used.
When done so, the pacakge will try to determine the biggest possible size of page
to use.
Package: bigquery-orm
python-bigquery-ormDocumentation:https://fernandocelmer.github.io/python-bigquery-ormSource Code:https://github.com/FernandoCelmer/python-bigquery-ormInstallationPIPpip install bigquery-ormPoetrypoetry add bigquery-ormUninstallationPIPpip uninstall bigquery-ormPoetrypoetry remove bigquery-ormCommit Style⚙️ FEATURE📝 PEP8📌 ISSUE🪲 BUG📘 DOCS📦 PyPI❤️️ TEST⬆️ CI/CD⚠️ SECURITYLicenseThis project is licensed under the terms of the MIT license.
Package: bigquery-sql-parser
Bigquery SQL ParserSimple and out of the box Bigquery SQL Parser for python.
Convert your SQL into python object which you can modify programatically.Installationpip3installbigquery-sql-parserUsageSince the SQL will be converted to Python object, hence the variable name will be very important to understand.Please see how we use the variable and terms below.QueryBelow is what we call syntax of query, andQueryis the object python name.#path/query.sqlSELECTid,nameAScustomer_name,FROMcustomer_tableAScustomerColumnid,customer_nameis columns and we have aColumnpython object for it.idis theColumn.valueas long as theColumn.name, butnameisColumn.valueandcustomer_nameis theColumn.nameGot the different ofvalueandnameofColumnobject? I hope so.Python sample:frombigquery_sql_parser.queryimportQueryquery=Query(your_query_here)columns=query.columnsforcolumninquery.columnsprint(column.value,column.name)#> id, id#> name, customer_nameTableYou can just get the table name by callingQueryObjectQuery.full_table_idsfrombigquery_sql_parser.queryimportQueryquery=Query(your_query_here)print(query.full_table_ids)CTEWell, if you use CTE like thisWITHcte1AS(xxxx),cte2AS(xxxx)SELECT*FROMcte2,cte1It can't usingQueryobject directly, you need to useScriptinsteadfrombigquery_sql_parser.scriptimportScriptfrombigquery_sql_parser.queryimportQueryscript=Script.from_file('query_path.sql')cte_list=script.ctesforcteincte_list:print(cte.name)print(cte.text)query=Query(cte.text)query...# any query variable can call hereBlocksSince BigQuery support scripting, sometime you might want to scan query that has a lot of blocks and use Semicolon as separator.CREATEFUNCTIONxAS(());DELETETABLEx;INSERTINTOx;WITHcte1AS(SELECT*FROMtable)SELECTaASb,cASdFROMtable_1It can't usingQueryobject directly, you need to useScriptinsteadfrombigquery_sql_parser.scriptimportScriptfrombigquery_sql_parser.queryimportQueryscript=Script.from_file('query_path.sql')blocks=script.blocksforblockinblocks:ifblock.is_cte_statementandblock.is_select_statement:print(cte.name)print(cte.text)cte_query=Query(cte.text)cte_query...final_query=Query(block.final_cte_query)final_query...And keep counting for others to supported...Where/Filter statement ObjectColumn Functions, Complex Column CompositionDML ObjectAdding context from BigQuery actual table like its partition, clusters, description, etcsWrite back from python Object to actual queryContributingPull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.Please make sure to update tests as appropriate.LicenseMIT
Package: bigrams
(N)GramsSimply create (N)grams: N ~ Bi | Tri ...Welcome to bigrams, a Python project that provides a non-intrusive way to connect tokenized sentences in (N)grams.
This tool is designed to work with tokenized sentences, and it is focused on a single task: providing an efficient way
to merge tokens from a list of tokenized sentences.It's non-intrusive as it leaves tokenisation, stopwords removal and other text preprocessing out of its flow.Source Code:https://github.com/proteusiq/bigramsPyPI:https://pypi.org/project/bigrams/Installationpipinstall-UbigramsUsageTo use bigrams, import it into your Python script, and usescikit-learn-ish API to transform your tokens.frombigramsimportGrams# expects tokenised sentencesin_sentences=[["this","is","new","york","baby","again!"],["new","york","and","baby","again!"],]g=Grams(window_size=2,threshold=2)out_sentences=g.fit_transform(in_stences)print(out_sentences)# [["this", "is", "new_york", "baby_again!"],#   ["new_york", "and", "baby_again!"],#  ]DevelopmentClone this repositoryRequirements:PoetryPython 3.7+Create a virtual environment and install the dependenciespoetryinstallActivate the virtual environmentpoetryshellTestingpytestPre-commitPre-commit hooks run all the auto-formatters (e.g.black,isort), linters (e.g.mypy,flake8), and other quality
checks to make sure the changeset is in good shape before a commit/push happens.You can install the hooks with (runs for each commit):pre-commitinstallOr if you want them to run only for each push:pre-commitinstall-tpre-pushOr if you want e.g. want to run all checks manually for all files:pre-commitrun--all-filesContributing are welcomeToDo:create a save & load functioncompare it with gensim Phraseswrite replacer in Rust -PyO3
Package: bigram-spam-classifier
# bigram-spam-classifier# A bigram approach for classifying Spam and Ham messages# install with pippip install bigram-spam-classifier# import in your python filefrom bigram_spam_classifier import spamclassifier# create an object of the classifier and pass your message as the parameterclassifier = spamclassifier.classifier("Customer service annoncement. You have a New Years delivery waiting for you. Please call 07046744435 now to arrange delivery")# classify the messagecls = classifier.classify()print(cls)# If returns 1 message is a Spam, if returns 0 message is a Ham# find the unigrams and bigrams in the messageunigrams = classifier.inputUnigramsprint(unigrams)bigrams = classifier.inputBigramsprint(bigrams)# find the bigram probabilities of Spam and Hamspam_probability = classifier.bigramPSpamprint(spam_probability)ham_probability = classifier.bigramPHamprint(ham_probability)
Package: bigraph
Bipartite link predictionPredicting links in bipartite networks on top of networkxNode based similarities and Katz has been implemented. you can find algorithms in prediction directory.
Algorithms untill now:NumberAlgorithm1.adamic adar2.common neighbors3.preferential attachment4.jaccard5.katz similarityHow to use the codeyou can run the code by placing your data in inputs folder and use predict methodes iplemented in "predict" file or directly call provided functions in the evaluation file.frombigraphimportbigraphasbgdefmain():"""Link prediction on a bipartite network:return: Predicted linkes"""df,df_nodes=import_files()G=make_graph(df)pr.aa_predict(G)# Here we have called Adamic Adar method from bigraph moduleor you can run evaluation methods directly which calls its peer method automaticallyfrombigraph.evaluationimportevaluationasevdefmain():"""Link prediction on a bipartite network:return: Predicted linkes"""df,df_nodes=import_files()G=make_graph(df)ev.evaluate(G,k=10,method='all')# Here we have evaluated all methods using evaluation module. Methods are 'jc', 'aa', 'pa', 'cn'MetricsMetrics that are calculated during evaluation:NumberEvaluattion metrics1.Precision2.AUC3.ROC4.returns fpr*5.returns tpr*For further usages and calculating different metricsDataset formatYour dataset should be in the following format:RowLeft side elementRight side elementWeight*1.ll0rl112.ll2rl113.ll1rl214.ll3rl315.ll4rl32Although the weight has not been involved in current version, but, the format will be the same.More examplesfrombigraphimportbigraphasbgdefmain():"""Link prediction on a bipartite network:return: Predicted linkes"""df,df_nodes=import_files()G=make_graph(df)pr.aa_predict(G)# Here we have called Adamic Adar method from bigraph modulepr.pa_predict(G)# Prefferencial attachmentpr.jc_predict(G)# Jaccard coefficientpr.cn_predict(G)# Common neighborsReferencesNumberReference1.Yang, Y., Lichtenwalter, R.N. & Chawla, N.V. Knowl Inf Syst (2015) 45: 751.https://doi.org/10.1007/s10115-014-0789-02.Liben-nowell, David & Kleinberg, Jon. (2003). The Link Prediction Problem for Social Networks. Journal of the American Society for Information Science and Technology.https://doi.org/58.10.1002/asi.205913....TODOModulate the functionsMake it faster using vectorization etc.Unify and reconstruct the architecture and eliminate redundancyDocumentationI will provide documentations whenever I could make time!:watch: or you can pull a request and help to make it happen togetherAfter running the main, it will export the graph in .json and .gexf format for furthur usages. For instance: Gephi etc.If it was helpful then hit the:star: