Saravanan666/pypi-org huggingface dataset:
Package: kez
A simple command line utility for tracking and building documents,
specificallyPelicanblogs.Usesclifffor the user interface,
together with a local sqlite database object-mapped withpeewee.Tested with Python-2.7 and Python-3.4. Unlikely to work on windows.UsageAdd a repository containing document source files:$ kez add myblog git@github.com:averagehuman/maths.averagehuman.org.gitBuild any documents defined therein:$ kez build myblogTrack projects:$ kez list
+---------+------------------------+---------+--------------------------------------------------------+
| Project | Document               | Type    | Url                                                    |
+---------+------------------------+---------+--------------------------------------------------------+
| myblog  | maths.averagehuman.org | PELICAN | git@github.com:averagehuman/maths.averagehuman.org.git |
+---------+------------------------+---------+--------------------------------------------------------+After building, if there is a rootindex.html, open the document in a
browser window:$ kez serve myblogConfigurationThe source repository must have an ini-style config file calledkez.cfgcontaining one or many sections, where each section defines
a particular document. The__docroot__value in each section
should give the directory, relative to the config file, where the
document sources are located (defaulting to the config file’s
directory).By convention, a double-underscored key is a build meta-option, while
any other key is an option required or with meaning to the program which
is called to produce the document (eg. Sphinx, Pelican,..).Examplekez.cfg[maths.averagehuman.org]
__doctype__ = pelican
AUTHOR = Buzz Lightyear (MSc Phd)
SITENAME = Beyond Infinity
SITEURL = blog.beyondinfinity.net
ARTICLE_URL = {date:%Y}/{date:%m}/{slug}/
ARTICLE_LANG_URL = {date:%Y}/{date:%m}/{lang}/{slug}/
PAGE_URL = {slug}/
PAGE_LANG_URL = {lang}/{slug}/
ARTICLE_SAVE_AS = {date:%Y}/{date:%m}/{slug}/index.html
ARTICLE_LANG_SAVE_AS = {lang}/{date:%Y}/{date:%m}/{slug}/index.html
PAGE_SAVE_AS = {slug}/index.html
PAGE_LANG_SAVE_AS = {lang}/{slug}/index.htmlSupported Document TypesPelicanIn the future, possiblySphinx.RequiredThe following libraries are required:cliffpelicanpeeweevcstoolsgiturlparse.pywatdarepopython-slugifyTypedInterpolationTestsRun tests withPython 2orPython 3:$ make test
$ make test PYVERSION=2
$ make test PYVERSION=3Python 3is the default if PYVERSION is not specified.
Package: kf6py
KF6 API for PythonThis is python package for calling KF6's API's, in hope to make it more friendly for learning analytic researchers.installpip install kf6pyusagefromkf6pyimportKF6API# we recommend using environment variables to store sensitive informationkf6api=KF6API(kfurl,username,password)# see the communities you are interested inkf6api.get_my_communities()# see the view in a particular communitykf6api.get_views("<community_id here>")# get the notes in a particular viewkf6api.get_notes_from_view("<community_id>","<view_id>")# create contribution in a viewkf6api.create_contribution("<community_id>","<view_id>",'<contribution_title>',"<contribution_content>")
Package: kfa
KFA[Google Colab]A fast Khmer Forced Aligner powered byWav2Vec2CTCandPhonetisaurus.Built-in Speech EnhancementWord-level AlignmentpipinstallkfaCLIkfa-aaudio.wav-ttext.txt-oalignments.jsonlPythonfromkfaimportalign,create_sessionimportlibrosawithopen("test.txt")asinfile:text=infile.read()y,sr=librosa.load("text.wav",sr=16000,mono=True)session=create_session()foralignmentinalign(y,sr,text,session=session):print(alignment)ReferencesMMS: Scaling Speech Technology to 1000+ languagesCTC FORCED ALIGNMENT API TUTORIALPhonetisaurusFine-Tune Wav2Vec2 for English ASR with 🤗 TransformersThai Wav2vec2 model to ONNX modelLicenseApache-2.0
Package: kfactory
KFactory 0.11.4Kfactory is agdsfactory-like tool. It is built withKLayoutas a backend instead of gdstk, but aims to offer the similar featuers.:exclamation:  KFactory is still experimental, expect API changes without notice (even though we try to keep it to a minimum!)It is suggest to pin the version of KFactory inrequirements.txtorpyproject.tomlwithkfactory==0.11.4for example.Features similar to gdsfactory:Cells & decorator for caching & storing cellsSimple routing (point to point and simpl bundle routes for electrical routes)Basic cells like euler/circular bends, taper, waveguidePath extrusion (no interface with CrossSections)Jupyter integrationPDK/package configurationPlugin system (simulations etc.) - CheckkpluginsGeneric PDK example - CheckkgenericNotable missing Features:CrossSectionNetlist/Schematics and LVSMore advanced routingNew/Improved Features:Fully hierarchical bi-directional conversion to YAMLAutomatic snapping to grid thanks to KLayoutMore features for vector geometries due to concept of Point/Edge/Vector/Polygon from KlayoutEasy booleans thanks to KLayout RegionsEnclosures: use the concept of enclosures, similar to cross sections, to allow automatic
calculation of boolean layers for structures based onminkowski sum,
which are built into KLayoutInstallationkfactory is available onpypipipinstallkfactoryAt the moment kfactory works only on python 3.10 and aboveDevelopment InstallationA development environment can be installed withpython-mpipinstall-e.[dev]It is defined inpyproject.toml. For committingpre-commitshould be installed withpre-commit install.
Package: kfda
Kernel FDAThis repository implements Kernel Fisher Discriminant Analysis (Kernel FDA) as described inhttps://arxiv.org/abs/1906.09436.
FDA, equivalent to Linear Discriminant Analysis (LDA), is a classification method that projects vectors onto a smaller subspace.
This subspace is optimized to maximize between-class scatter and minimize within class scatter, making it an effective classification method.
Kernel FDA improves on regular FDA by enabling nonlinear subspaces using thekernel trick.FDA and Kernel FDA classify vectors by comparing their projection in the fisher subspace to class centroids, adding a new class is just a matter of adding a new centroid.
Thus, this model is implemented here with the hope of using Kernel FDA as a oneshot learning algorithm.UsageKfdausesscikit-learn's interface.Initializing:cls = Kfda(n_components=2, kernel='linear')for a classifier that a linear kernel with 2 components.
For kernel of degree 2, useKfda(n_components=2, kernel='poly', degree=2)for a polynomial kernel of degree 2.
Seehttps://scikit-learn.org/stable/modules/metrics.html#polynomial-kernelfor a list of kernels and their parameters, or thesource code docstringsfor a complete description of the parameters.Fitting:cls.fit(X, y)Prediction:cls.predict(X)Scoring:cls.score(X, y)ExamplesSeeexamplesfor examples on MNIST, faces, and oneshot learning.After running them, you can plug corresponding pairs of generated*embeddings.tsvand*labels.tsvinto Tensorflow'sEmbedding Projectorto visualize the embeddings.
For example, runningmnist.pyand then loadingmnist_test_embeddings.tsvandmnist_test_labels.tsvshows the
following using the UMAP visualizer:NotebookAnother place to see example usage is theColab Notebook.CaveatsSimilar to SVM, the most glaring constraint of KFDA is the memory limit in training.
Training a Kernel FDA classifier requires creating matrices that aren_samplesbyn_sampleslarge, meaning the memory requirement grows with respect toO(n_samples^2).The accuracy, while high (0.97 on MNIST), seems to be limited by the training set size.
With a training size of 10000 and a testing size of 60000, performance on MNIST averages around 0.97 accuracy using 9 fisher directions and the RBF kernel:cls=Kfda(kernel='rbf',n_components=9)This may be due to the constrained training size.
Accuracy can be improved without increasing training size by implementing invariant kernels that would implicitly handle scale and rotation without requiring an extended dataset.Oneshot LearningOneshot learning means that an algorithm can learn a new class with as little as one sample.
This is possible for Kernel FDA because it finds a subspace that purposefully spreads out distinct classes.
Introducing a new label involves simply adding another centroid for use in prediction.
See theColab Notebook.
or theexamplefor examples.
Package: kfg
kfg: A configuration library for Pythonkfgprovides aConfigclass which contains your program’s configuration
data. It lets you access, manipulate, and validate this data in a
straightforward way. It also provides a means to de/serialize the data.Basic usageSetting and retrieving configuration valuesGetting and setting configuration values is simple:fromkfg.configimportConfig# construct a Configc=Config()# set the ('ui', 'font-color') value to 'blue'c['ui','font-color']='blue'# retreive the valuefont_color=c['ui','font-color']Uniform exceptionskfgprovides a uniform set of exceptions that are used to signal configuration
problems. This means that your program can catch these exceptions and know that
they only ever indicate configuration problems. For example:c=Config()# kfg throws ConfigKeyError on missing keys.try:x=c['foo']exceptConfigKeyError:print('foo not in the config')Transforms and validationkfglets you associate a “transform” with a key. This transform is a 1-arity
callable that will be passed the stored value of the configuration option, andkfgwill pass the return value of the transform to the user when they access
the value. If a transform fails,kfgwill raise aConfigValueError.This lets you do two things. First, you can construct arbitrary values from
stored configuration information in a centralized way, i.e. mediated by the
configuration. Second, it let’s you validate configuration values. For example:c=Config()c['processing','timeout']="10 seconds"c.set_transform(('processing','timeout'),float)c['processing','timeout']# Raises ConfigValueError because ``float``# can't parse "10 seconds"This system is intentionally low-powered and simple. For example,kfgallows
you tosetvalues which violate the transform. The goal is not to create
completely water-tight configurations, but rather to create systems which are
easy to get right and which warn you when you might be using an invalid value.RationaleThere’s no real rocket science inkfg, and you can get most of its features
just by using dictionaries, lists, tuples, etc. There are a few problems with
using “raw” data structures like that for configuration, though.First, you’ll get standard exceptions likeKeyErrorandIndexErrorwhen you
try to access missing values. Since these kinds of errors can come from almost
anywhere in a system, it’s not easy to differentiate between those that come
from configuration problems and the others. By providing specialized
“configuration” errors, you can catchkfgexceptions and be confident that
they point to configuration errors.Second,kfglets you centralize the basic configuration
validation/transformations. Configuration values may be used in many places in a
system, so it’s often helpful to have a single point of validation for them.
Package: kfinny.cachedvt
cached-virustotal-apiA cached extension of the officialPython clientfor VirusTotal.UsageCreate the CachedClient like so:fromkfinny.cachedvtimportCachedClientclient=CachedClient(<apikey>,cache_dir=<somefolder>)The folder does not need to exist, but if it does, it should be empty or a prior cache from this library. The cache is
sqlite database created byDiskCache.The client may be used just as it is documented by thevt-pyproject.
The codebase "adds" a single function to the client API,yield_file_reportand can take a list of hashes.# yield_file_reporthashes=[<listofmd5,sha1,sha256hashes>]forobjinclient.yield_file_report(hashes):assert(obj.type=='file')print(f'{obj.md5}:{obj.sha1}:{obj.sha256}')Files not found on VirusTotal are not returned, but they are cached, since the whole point of this library is to conserve
quota.>>>hits,misses=client.cache.stats()>>>reports=list(client.yield_file_report('abcd'*16))>>>reports[]>>>_hits,_misses=client.cache.stats()>>>assert(_misses-misses==1)>>>Prior VersionsAprevious versionof this code worked with thevirustotal-apilast tested using this commit551b87a.
Package: kfjsonfilter
KF pubsubJsonFieldFilter and JsonFieldSearchFilter implementation with django and django rest frameworkInstallationWith requirements.txtAdd the following line in yourrequirements.txtfile:kfjsonfilterWith PIP$pipinstallkfjsonfilterGetting StartedThere are implemented two base filters: JsonFieldFilter and JsonFieldSearchFilter.JsonFieldFilterAllow create a data filter from a json object. Its neccesary to declare two elements in the django rest framework viewset: json_field and json_filtersTYPEDESCRIPTIONjson_fieldstringIndicates the json field object form the modeljson_filterslist of dictsIndicates the field thats allow filter and lookup type for each typejson_filters structureAllows filter by any value from a json data. Each element must be included in a dictionary with to key: field and lookup.json_filters = [{'field': <field_name>, 'lookup': <lookup>}, ...]DESCRIPTIONfieldfield_name is the name of the element to filter included in json datalookuplookup is the filter type, icontains, iexact, in, etc... and date_range (to filter by a date range). to filter only a date by a date indicates only lte, gte, exact...JsonFieldSearchFilterAllow create a data filter and search filter from a json object. Its neccesary to declare three elements in the django rest framework viewset: json_field and search_fieldsTYPEDESCRIPTIONjson_fieldstringIndicates the json field object form the modeljson_filterslist of dictsIndicates the field thats allow filter by a field and lookup type for each typesearch_fieldslist of dictsIndicates the field thats allow search by a field and lookup type for each typejson_filters and search_fields structureAllows filter/search by any value from a json data. Each element must be included in a dictionary with to key: field and lookup.json_filters = [{'field': <field_name>, 'lookup': <lookup>}, ...]DESCRIPTIONfieldfield_name is the name of the element to filter included in json datalookuplookup is the filter type, icontains, iexact, in, etc... and date_range (to filter by a date range). to filter only a date by a date indicates only lte, gte, exact...Also, when lookup is 'date_range' field_name must finish with []ExampleDjango Modelclass MyModel(models.Model):
	data = JSONField(default=dict)For example:
data = {'creation_date': '2020-01-01', 'total': 3, 'name': 'Pedro Miguel'}Django Rest Framework Viewclass MyModelViewSet(ListAPIView):
	queryset = MyModel.objects.all()
	filter_backends = [JsonFieldFilter, DjangoFilterBackend, OrderingFilter]
	json_field = 'data'
	json_filters = [{'field': 'creation_date', 'lookup': 'date_range'}, {'field': 'total', 'lookup': 'gte'}]

or

class MyModelViewSet(ListAPIView):
	queryset = MyModel.objects.all()
	filter_backends = [JsonFieldSearchFilter, DjangoFilterBackend, OrderingFilter]
	json_field = 'data'
	json_filters = [{'field': 'creation_date', 'lookup': 'lte'}, {'field': 'total', 'lookup': 'gte'}]	
	search_fields = [{'field': 'name', 'lookup': 'icontains'}]
Package: kfk2es
kafka to elasticsearch从kafka读取数据，并处理后写入elastcsearch,只支持python3使用方法编写自己的处理程序example.pyfromkfk2esimportStreamProcessdefyour_own_handler(event):"""event的内容就是从kafka中获取的每条数据"""print(event)#return ('esindex-%Y%m%d', '_log', event)#return (None, '_log', event)returnevent# 返回的结果会发到es中，如果没有返回，则不发if__name__=='__main__':pipe=StreamProcess(configfile='config.yml')# from myconf import Conf# conf = Conf('config.yml')# pipe = StreamProcess(configfile=conf)pipe.handler=your_own_handlerpipe.run()StreamProcess的configfile参数可以是一个配置文件路径，也可以是包含必要参数的字典。如果不添加任何处理程序，那么从kafka读取的数据会直接尝试往elasticsearch发送自己添加处理程序，只需要编写函数，并按照如下格式进行返回:返回一个字典， 那么字典中的数据会被传到es中返回一个3个元素的tuple， 那么第一个元素是es的索引，第二个元素则是es的类型，第三个元素为需要导入es的数据。
通过数据直接返回的索引和类型要优先于配置文件中的配置。如果只希望动态设置其中一个，另一个使用默认，那么不需要定义的设置为None即可如果不需要传输到es中，只要不返回任何数据就可以了。这样可以只测试kafka的连通性编辑配置文件配置文件可以是json或者yaml格式，以json,yaml或者yml结尾可以使用多个kafka作为输入，elasticsearch只有一个输出yaml格式例子config.ymlkafka:-topic:topbootstrap_servers:-server1:9092-server2:9092-server3:9092group_id:group_idsasl_plain_username:usersasl_plain_password:passwordelasticsearch:params:#parms for elasticsearchhosts:-http://user:pass@10.1.0.1:9200index:test-%Y%m%dtype:log# optional(default: "log")cache_size:150# optional(default: 150)timeout:1# optional(default: 1)需要注意的是elasticsearch选项中的parms选项包含了elasticsearch库中Elasticsearch的参数. 而其他参数是StreamProcess需要的
Package: kforge
KForge is an enterprise software application for project hosting. KForge provisions project services on demand. KForge controls access with a robust, role-based, single sign-on access controller.
KForge project services may include version control systems (e.g. Git, Mercurial, Subversion), project frameworks with mechanisms to plan and track work (e.g. Trac), wikis and mailing lists (e.g. MoinMoin, Mailman), and content management systems and blogs (e.g. Joomla, Wordpress).
Package: kforgeinstall
Welcome to the KForge InstallerThe KForge installer can be used to install and setupKForgeservices.Install GuideInstall the KForge installer either by running:$ pip install kforgeinstallOr by downloading the KForge installer tarball, unpacking and running:$ python setup.py installAfter installation, please read thekforge-installhelp page for usage information:$ kforge-install --help
Package: kfp
Kubeflow Pipelines is a platform for building and deploying portable, scalable machine learning workflows based on Docker containers within theKubeflowproject.Use Kubeflow Pipelines to compose a multi-step workflow (pipeline) as agraphof containerizedtasksusing Python code and/or YAML. Then,runyour pipeline with specified pipeline arguments, rerun your pipeline with new arguments or data,scheduleyour pipeline to run on a recurring basis, organize your runs intoexperiments, save machine learning artifacts to compliantartifact registries, and visualize it all through theKubeflow Dashboard.InstallationTo installkfp, run:pipinstallkfpGetting startedThe following is an example of a simple pipeline that uses thekfpv2 syntax:fromkfpimportdslimportkfp@dsl.componentdefadd(a:float,b:float)->float:'''Calculates sum of two arguments'''returna+b@dsl.pipeline(name='Addition pipeline',description='An example pipeline that performs addition calculations.')defadd_pipeline(a:float=1.0,b:float=7.0,):first_add_task=add(a=a,b=4.0)second_add_task=add(a=first_add_task.output,b=b)client=kfp.Client(host='<my-host-url>')client.create_run_from_pipeline_func(add_pipeline,arguments={'a':7.0,'b':8.0})
Package: kfp-deployer
kfp-deployerDeploy your ml-pipeline withkfp-deployfrom cli.How to usekfp-deploy https://your-kubeflow-host/ "pipeline-name" ./pipeline_file.yamlfor more detail, seekfp-deploy -h.what the difference fromkfp pipeline upload?Kubeflow Pipelines requires the all pipelines must have unique names.
Otherwise you have to useupdate the versioninstead ofupload the pipeline.
Furthermore, you have to use unique name when uploading the new version of pipeline.This command does everything required in the upload process for you.
This command will communicate with kfp host and automatically determine
whether update or upload is required and perform it.
Furthermore, the version string is automatically generated based on the upload timestamp.
Package: kfpdist
Kubeflow Pipeline distributed training supportkfp-dist-train contains utilities to use together withKubeflow Pipelineto enable writing distributed training code directly using Kubeflow Pipeline SDK.Get StartedSetup an Kubeflow environment (maybe usehttps://github.com/alauda/kubeflow-chart).Upload the examplekfp-dist-train.ipynbinto a Notebook
instance, or setup local pipeline submit.Execute the example to submit a workflow, you can configure the number of workers
in the Kubeflow web UI. The job should look like below:Some Roadmapsupportkfpdist.component(dist=True)decorator as an wrap ofdsl.componentsupport parameter server strategysupport pytorch
Package: kfpga
kFPGAkFPGA is an opensource platform for creating and programming FPGA cores.InstallationUsing pippython3 -m pip install kfpgaFrom sourcegit clone https://git.slaanesh.org/killruana/kfpga.git
python3 kfpga/setup.py installUsageCreating a new kFPGA coreFirst, create the kFPGA corekfpga-creator createcore \
    --width 5 \
    --height 5 \
    --io 4 \
    --clocks 1 \
    --sets 1 \
    --resets 1 \
    --enables 1 \
    --interconnect_width 10 \
    --le 4 \
    --lut 4 \
    kFPGADemoCoreThen, you can generate the RTL of the core like thiskfpga-creator generatertl \
    kFPGADemoCore/kFPGADemoCore.kcpImplementationkFPGA cores can be implemented on FPGA for testing purpose, as an ASIC or integrated in a SoC.TODOProgrammingkFPGA cores are programmed with thekfpga-programmercommandTODOArchitectureCurrently, the architecture of kFPGA core is very simple: only LUTs and interconnect. The following functionalities are planned:DSP blocksmemory blockscarry chainsupport for generated clock, set, reset and enable signals
Package: kfp-kubernetes
Kubernetes Platform-specific FeaturesThekfp-kubernetesPython library enables authoringKubeflow pipelineswith Kubernetes-specific features. These features are supported by thedefault KFP open source BE. Specifically, thekfp-kuberneteslibrary supports authoring pipelines that use:SecretsPersistentVolumeClaimsSee thekfp-kubernetesreference documentation.InstallationThekfp-kubernetespackage can be installed as akfpSDK extra dependency withkfp==2.x.x:pipinstallkfp[kubernetes]--preOr installed independently:pipinstallkfp-kubernetesExample usageSecret: As environment variablefromkfpimportdslfromkfpimportkubernetes@dsl.componentdefprint_secret():importosprint(os.environ['my-secret'])@dsl.pipelinedefpipeline():task=print_secret()kubernetes.use_secret_as_env(task,secret_name='my-secret',secret_key_to_env={'password':'SECRET_VAR'})Secret: As mounted volumefromkfpimportdslfromkfpimportkubernetes@dsl.componentdefprint_secret():withopen('/mnt/my_vol')asf:print(f.read())@dsl.pipelinedefpipeline():task=print_secret()kubernetes.use_secret_as_volume(task,secret_name='my-secret',mount_path='/mnt/my_vol')PersistentVolumeClaim: Dynamically create PVC, mount, then deletefromkfpimportdslfromkfpimportkubernetes@dsl.componentdefmake_data():withopen('/data/file.txt','w')asf:f.write('my data')@dsl.componentdefread_data():withopen('/reused_data/file.txt')asf:print(f.read())@dsl.pipelinedefmy_pipeline():pvc1=kubernetes.CreatePVC(# can also use pvc_name instead of pvc_name_suffix to use a pre-existing PVCpvc_name_suffix='-my-pvc',access_modes=['ReadWriteOnce'],size='5Gi',storage_class_name='standard',)task1=make_data()# normally task sequencing is handled by data exchange via component inputs/outputs# but since data is exchanged via volume, we need to call .after explicitly to sequence taskstask2=read_data().after(task1)kubernetes.mount_pvc(task1,pvc_name=pvc1.outputs['name'],mount_path='/data',)kubernetes.mount_pvc(task2,pvc_name=pvc1.outputs['name'],mount_path='/reused_data',)# wait to delete the PVC until after task2 completesdelete_pvc1=kubernetes.DeletePVC(pvc_name=pvc1.outputs['name']).after(task2)