Saravanan666/pypi-org huggingface dataset:
Package: tf-reaper
ThreshingFloor CLI Quickstart Guide================================The ThreshingFloor CLI helps you separate the signal from the noise in your logfiles. If you are running a service that faces the internet, you likely see thousands of scans, bots, and brute force attempts every day. These scans clog up your log files, and make it hard to find legitimate events of interest.The ThreshingFloor CLI is a utility that leverages the ThreshingFloor API to reduce noisy entries from your log files. This tool is currently in ALPHA.
Package: tfrecord-dataset
TFRecord reader, writer, and PyTorch DatasetThis library allows reading and writingTFRecordfiles efficiently in Python, and provides anIterableDatasetinterface for TFRecord files in PyTorch. Both uncompressed and compressed gzip TFRecord are supported.This library is modified fromtfrecord, to remove its binding totf.Exampleand support generic TFRecord data.Installationpipinstalltfrecord-datasetUsageBasic read & writeimporttfrecord_datasetastfrwriter=tfr.TFRecordWriter('test.tfrecord')writer.write(b'Hello world!')writer.write(b'This is a test.')writer.close()forxintfr.tfrecord_iterator('test.tfrecord'):print(bytes(x))TFRecordDatasetUseTFRecordDatasetto read TFRecord files in PyTorch.importtorchfromtfrecord_dataset.torchimportTFRecordDatasetdataset=TFRecordDataset('test.tfrecord',transform=lambdax:len(x))loader=torch.utils.data.DataLoader(dataset,batch_size=2)data=next(iter(loader))print(data)Sharded TFRecordsThe followingTFRecordDatasetreads TFRecord data from 8 files in parallel. The name of these 8 files match patterndata-0000?-of-00008.dataset=TFRecordDataset(data@8', transform=lambda x: len(x))Data transformationThe reader reads TFRecord payload as bytes. You can pass a callable as thetransformargument for parsing the bytes into the desired format, as
shown in the simple example above. You can use such transformation for parsing
serialized structured data, e.g. protobuf, numpy arrays, images, etc.Here is another example for reading and decoding images:importcv2dataset=TFRecordDataset('data.tfrecord',transform=lambdax:{'image':cv2.imdecode(x,cv2.IMREAD_COLOR)})Shuffling the dataTFRecordDatasetautomatically shuffles the data with two mechanisms:It reads data into a buffer, and randomly yield data from this buffer. Setting to buffer to a larger size (buffer_size) produces better randomness.For sharded TFRecords, it reads multiple files in parallel. Settingfile_parallelismto a larger number also produces better randomness.IndexIndex files are deprecated since v0.2.0. It's no longer required.Such index files can be generated with:python -m tfrecord_dataset.tools.tfrecord2idx <tfrecord path> <index path>Infinite and finite datasetBy default,TFRecordDatasetis infinite, meaning that it samples the data forever. You can make it finite by settingnum_epochs.dataset=TFRecordDataset(...,num_epochs=2)AcknowledgementsThis repo is forked fromhttps://github.com/vahidk/tfrecord.
Package: tfrecordutils
# TFRecordUtilsMaking life easier for working with TFRecord files and TensorFlow's data pipeline API.## Installation```pip install tfrecordutils```### HOWTO### Create a TFRecord file from a directory of images* crops images by default to 500x500, but you can override this```jpg_glob     = './jpg/*.jpg'tfrecord_file = 'images.tfrecords'TFRecordUtils.jpgToTFRecord(jpg_glob)```### HOWTO### Get an image batch iterator from a TFRecord file```tfrecord_file = 'images.tfrecords'with tf.Session() as sess:iterator = TFRecordUtils.get_image_batch_iterator(sess, tfrecord_file)batch = sess.run(iterator)print(batch.shape)```
Package: tfreecord
TFReecordRead & Write TFRecord without heavy Tensorflow.pip3installtfreecordExampleWrite TFRecordimporttfreecordwriter=tfreecord.RecordWriter()feature={'id':writer.int64_feature(759),'text':writer.bytes_feature("Now rides our knight".encode("utf-8")),'label':writer.bytes_feature("poem".encode("utf-8")),}withopen("poems.tfrecord","ab")asf:f.write(writer.encode_example(feature))Read TFRecordimporttfreecordreader=tfreecord.RecordReader()fordatainreader.read_from_tfrecord("poems.tfrecord"):print(data)Thanks toJong Wook Kimblog post about TFRecordshereand histfrecord_litelibrary.
Package: tf-restore-helper
FeaturesConstructs Terraform import commands to assist an operator aligning a terraform state after volume restores have taken place by an external tool, which results in terraform misalignment.Tool is currently only compatible with AWS volumes (PRs welcome) and is used at own risk. Operators that do not understand the terraform state file or what the produced commands will do, should not use this tool as a helper and proper state backup and recovery procedures should be well understood before making use of this.InstallationYou can installTf Restore HelperviapipfromPyPI:$pipinstalltf-restore-helperUsagePlease see theCommand-line Referencefor details.Examples of usage:> tf-restore-helper --planfile tests/assets/plan.json
 2021-05-03 13:28:43 sbpltc1nplvdl botocore.credentials[79705] INFO Found credentials in environment variables.
 2021-05-03 13:28:46 sbpltc1nplvdl postrestoretfcli[79705]   WARNING THIS INFORMATION SHOULD BE USED ONLY IF YOU KNOW WHAT YOU ARE DOING!
 -----------Terraform volume alignment commands-----------
 terraform state rm "aws_ebs_volume.test-instance-volume-1"
 terraform state rm "aws_volume_attachment.test-instance-volume-1"ContributingContributions are very welcome.
To learn more, see theContributor Guide.LicenseDistributed under the terms of theMIT license,Tf Restore Helperis free and open source software.IssuesIf you encounter any problems,
pleasefile an issuealong with a detailed description.CreditsThis project was generated from@cjolowicz’sHypermodern Python Cookiecuttertemplate.
Package: tfr-image
TFRimageAn experimental minimal tool to create TFRecords from asmall datasetof images (few thousand images max). Based on code fromcreate_tfrecordsbut adapted to Tensorflow 2.0 with some changes.📋 Installationpipinstalltfr_image💻 UsageExample of the directory structure:├── cat_dogs_sample
│   │── train
│       │── cat
│            │── cat1.jpg
│            │── cat2.jpg
│       │── dog
│            │──  dog1.jpg
│            │── dog2.jpgExample to createtfrecordsinsidedataset_dirdirectoryfromtfr_imageimportTFRimagetool=TFRimage()tool.create_tfrecords(dataset_dir="../cat_dogs_sample/train",tfrecord_filename="cat_dogs",validation_size=0.2,num_shards=2,)Other Toolstensorflow-recorder: Google open source tool forBigdataset of images that provides connectivity with Google Cloud Dataflow.
Package: tfrmaker
tfrmakerUtility package which helps to ease the manipulation of tfrecords.ContentsDescriptionInstallationUsageSupportContributeLicenseDescriptiontfrmakerhelps to ease the manipulation of tfrecords for your next machine learning project withtensorflow. You can now easily create, extract and load image datasets in the form of tfrecords with help oftfrmaker. With the help of the package, large image datasets can be converted into tfrecords and fed directly into tensorflow models for training and testing purposes. Some key feature of the package includes:dynamic resizingsplitting tfrecords into optimal shardsspliting training, validation, testing of tfrecordscount no of images in tfrecordsasynchronous tfrecord creationWhy TFRecords?TFRecords stores data as a sequence of binary records with help ofprotocol buffers, a cross-platform, cross-language library. It has many advantages like:Efficient storage: TFRecord data can take up less space than the original data; it can also be partitioned into multiple files.Fast I/O: TFRecord format can be read with parallel I/O operations, which is useful for TPUs or multiple hosts.InstallationUse the package managerpipto install tfrmaker.pipinstalltfrmakerUsageA minimal usage oftfrmakerwith image data, organized as directores with name as class labels:fromtfrmakerimportimages,display# mapping label names with integer encoding.LABELS={"bishop":0,"knight":1,"pawn":2,"queen":3,"rook":4}# specifiying data and output directories.DATA_DIR="datasets/chess/"OUTPUT_DIR="tfrecords/chess/"# create tfrecords from the images present in the given data directory.images.create(DATA_DIR,LABELS,OUTPUT_DIR)# load one or more tfrecords as an iterator object.dataset=images.load(["tfrecords/chess/queen.tfrecord","tfrecords/chess/bishop.tfrecords"],batch_size=32,repeat=True)# iterate one batch and visualize it along with labels.databatch=next(iter(dataset))display.batch(databatch,LABELS)Referexamplesfolder for more advanced usage.Support"Your mental support by staring the repo is much appreciated."ContributePull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.Please make sure to update tests as appropriate.LicenseMIT
Package: tfrq
tfrq - an easy way to parallelize processing a functiontfrq on github!Stop waiting for your code to finish, start using tfrq - the effortless solution for parallelizing your functions and supercharging your performance!This library provides an easy way to parallelize the execution of a
function in python using the concurrent.futures library. It allows you
to run multiple instances of a function simultaneously, making your code
run faster and more efficiently. It also provides a simple API for
managing the process, allowing you to cancel or wait for the completion
of a task. With this library, you can easily take advantage of the power
of parallel processing in python.Here’s an example of how you can use the library to parallelize the
execution of theprintfunction:Example 1:from tfrq import tfrq
params = ["Hello", "World", "!"]
func = print
tfrq(func=func, params=params, num_cores=3)Example 2:input_list = [[1, 2], [3, 4], [5, 5], [6, 7]]
list_of_results_for_all_pairs = tfrq(sum, input_list)
print(list_of_results_for_all_pairs)  # [[3], [7], [10], [13]] -- result for each pair ordered.This code will call thesumfunction in parallel with the given
parameters and use all cores, so it will print the given parameters in
parallel.Example 3 - using the config parameter:input_list = [[1, 2], [3, 4], [5, 5], [6, str(7) + '1']]  # error in final input
list_of_results_for_all_pairs = tfrq(sum, input_list)
print(list_of_results_for_all_pairs)  # [[3], [7], [10], []] -- result for each pair ordered.

input_list = [[1, 2], [3, 4], [5, 5], [6, str(7) + '1']]  # error in final input
list_of_results_for_all_pairs = tfrq(sum, input_list, config={"print_errors": True})
# unsupported operand type(s) for +: 'int' and 'str'
print(list_of_results_for_all_pairs)  # [[3], [7], [10], []] -- result for each pair ordered.

input_list = [[1, 2], [3, 4], [5, 5], [6, str(7) + '1']]  # error in final input
list_of_results_for_all_pairs, errors = tfrq(sum, input_list,
                                             config={"print_errors": True, "return_errors": True})
# unsupported operand type(s) for +: 'int' and 'str'
print(list_of_results_for_all_pairs)  # [[3], [7], [10], []] -- result for each pair ordered.
print(errors)  # [[], [], [], [TypeError("unsupported operand type(s) for +: 'int' and 'str'")]]Example 4 - operator to apply on parameters:operator=None  -> func(args)
operator="*"   -> func(*args)
operator="**"  -> func(**args)

params = ["Hello", "World", "!"]
func = print
tfrq(func=func, params=params, num_cores=3, operator="*")
# H e l l o
# !
# W o r l d ---- notice now it is func(*args) - that is causing the spaces.

params = ["Hello", "World", "!"]
func = print
tfrq(func=func, params=params, num_cores=3)
# Hello
# World
# !default config:config = {"return_errors": False, "print_errors": True}tfrq is an arabic word meaning “To Split”, which is the purpose of this
simple method, to split the work of a single function into multiple
processes as easy as possible.
Package: tf-sdp
Deep nonparametric conditional discrete probability estimation via smoothed dyadic partitioning.See thetfsdpdirectory (specifically models.py) for the details of all the models we implemented. Note that this file contains a lot of garbage code and legacy naming that needs to be cleaned up. Our SDP model is named LocallySmoothedMultiscaleLayer and is often referred to in some of the experiments as trendfiltering-multiscale.See theexperimentsdirectory for the code to replicate our experiments.Note that you should be able to install the package as a local pip package viapip -e .in this directory. The best example for how to run the models is inexperiments/uci/main.py, which contains the most recent code and should not have any API issues.CitationIf you use this code in your work, please cite the following:
Package: tf-sentence-transformers
tf-sentence-transformersA simple tensorflow/keras wrapper aroundsentence-transformers. The main class isSentenceTransformer, akeras.Layerthat can be constructed from a pretrained checkpoint in huggingface model hub.Example usage:from tf_sentence_transformers import SentenceTransformer

layer = SentenceTransformer.from_pretrained(
    "sentence-transformers/all-MiniLM-L6-v2"
)
inputs = [
    ["This is a test sentence."],
    ["This is another test sentence."],
]
embedding = layer(inputs)
# assert embedding.shape == (2, 384)
Package: tfseqestimator
tfseqestimatorSequence estimators for TensorFlow.Available estimatorsFullSequenceClassifier: one class for a whole sequenceFullSequenceRegressor: one value for a whole sequenceSequenceItemsClassifier: one class for each sequence itemSequenceItemsRegressor: one value for each sequence itemUsagefromtfseqestimatorimportFullSequenceClassifier,RnnTypeimporttensorflow.contrib.feature_columnascontrib_featurestoken_sequence=contrib_features.sequence_categorical_column_with_hash_bucket(...)token_emb=contrib_features.embedding_column(categorical_column=token_sequence,...)estimator=FullSequenceClassifier(sequence_feature_columns=[token_emb],rnn_type=RnnType.REGULAR_STACKED_LSTM,rnn_layers=[32,16])# Input buildersdefinput_fn_train:# returns x, ypassestimator.train(input_fn=input_fn_train,steps=100)definput_fn_eval:# returns x, ypassmetrics=estimator.evaluate(input_fn=input_fn_eval,steps=10)definput_fn_predict:# returns x, Nonepasspredictions=estimator.predict(input_fn=input_fn_predict)
Package: tfserver
Introducetfserveris an example for servinggRPCfor Tensorflow/Pytorch thing’s models.It can serve not only throughgRPCbut alsoRESTful APIwithSkitai App EngineandAtilaWSGI container.This project is inspired byissue #176.I’m so sorry about this soulless manual.Installationfrom version 0.3, it is now TensorFlow 2+ compatible (deprecated, no longer support)from version 0.4, TensorFlow 2+ onlyBuilding Model and DeployPlease seehttps://gitlab.com/hansroh/skitai/-/blob/master/tests/level4-2/build_model.pyIt is mostly used tensorflow keras anddnn.Creating Your Own gRPC ServerPlease seehttps://gitlab.com/hansroh/skitai/-/blob/master/tests/examples/tfserve.pyYou can know how to serve gRPC service and make yoyr own APIs.APIsPlease seehttps://gitlab.com/hansroh/tfserver/-/blob/master/tfserver/export/skitai/__export__.pyAPIs to manage models and basic inference.And for usage see,https://gitlab.com/hansroh/skitai/-/blob/master/tests/level4-2/test_tfserver.pyRelease History0.4 (2021. 4)upgrade for tensorflow 20.3 (2020. 6. 28)add model management APIsreactivate project and compatible with TF2+0.2 (2020. 6. 26): integrated with dnn 0.30.1b8 (2018. 4. 13): fix grpc trailers, skitai upgrade is required0.1b6 (2018. 3. 19): found works only grpcio 1.4.00.1b3 (2018. 2. 4): add @app.umounted decorator for clearing resource0.1b2: remove self.tfsess.run (tf.global_variables_initializer())0.1b1 (2018. 1. 28): Beta release0.1a (2018. 1. 4): Alpha release
Package: tfset
Change the hyper-parametersof your Tensorflow training session on
the fly. The package allows you to schedule events that change the
values of arbitrary Tensors with a simple command.RequirementsPython >= 3tensorflow >= 1.0Set UpInstall the package with pip:pip install tfsetOr clone and install from github:gitclonehttps://github.com/ondrejba/tfset.gitcdtfsetpythonsetup.pyinstallUsagetfset DEMOCheckMNIST_demo.ipynbfor a demostration of the usage of tfset in a simple
training script.ServerImport tfset server.import tfset.server as serverCreate Tensors for your hyper-parameters.learning_rate = tf.get_variable("learning_rate", initializer=tf.constant(0.1, dtype=tf.float32))
dropout_prob = tf.get_variable("dropout_prob", initializer=tf.constant(0.9, dtype=tf.float32))Create and start a Session Server.# "session" is a Tensorflow session
s, thread = server.run_server([learning_rate, dropout_prob], session)Periodically check for events.# "step" is the global step of your training procedure
s.check_events(step)Stop the server.s.shutdown()
thread.join(timeout=10)ClientGet status.tfset-sAdd an event (this event sets the learning rate to 0.01 at iteration
10000).tfset-a-nlearning_rate:0-i10000--value0.01Remove an event (with index 0 in this case).tfset-r-e0Eventstfset schedules hyper-parameter changes based onevents. An event contains the following information:iteration: when to execute the eventTensor name: which Tensor to changevalue: value to set the Tensor toThe reason for the use of events is that you might want to schedule
hyper-parameter changes in the future (e.g. lower learning rate to 10e-3
at 800k iteration). If two events targeting the same Tensor are
scheduled at the same iteration, the one that was scheduled later is
going to be executed.
Package: tf-settransformer
Set Transformer FrameworkAn unofficial implementation of the Set Transformer framework in Tensorflow/Keras based on the paper:Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks.The official pytorch repository is available at:https://github.com/juho-lee/set_transformer.InstallationThe set transformer framework can be installed by executing the following command:pipinstalltf-settransformer
Package: tf-siammask
tf-siammaskSiamMaskimplementation with Tensorflow 2.Installpipinstalltf-siammaskExampleimportnumpyasnpimportPIL.Imageimportsiammasksm=siammask.SiamMask()# Weight files are automatically retrieved from GitHub Releasessm.load_weights()# Adjust this parameter for the better mask predictionsm.box_offset_ratio=1.5img_prev=np.array(PIL.Image.open('data/cat1.jpg'))[...,::-1]box_prev=np.array([[227,184],[381,274]])img_next=np.array(PIL.Image.open('data/cat2.jpg'))[...,::-1]# Predicted box and mask images is created if `debug=True`box,mask=sm.predict(img_prev,box_prev,img_next,debug=True)Test dataPrevious frameNext frameFile name./data/cat1_with_box.jpg./data/cat2.jpgImagePredicted mask for./data/cat2.jpgTODOBounding-box regressionMask refinement networkPre-trained model for Tensorflow 2.0Training codeObject tracking codeReference@inproceedings{wang2019fast,
    title={Fast online object tracking and segmentation: A unifying approach},
    author={Wang, Qiang and Zhang, Li and Bertinetto, Luca and Hu, Weiming and Torr, Philip HS},
    booktitle={Proceedings of the IEEE conference on computer vision and pattern recognition},
    year={2019}
}
Package: tf-simple-metric-learning
Simple metric learning via tf.kerasThis package provides only a few metric learning losses below;ArcFaceAdaCosCircleLossI have been greatly inspired byPyTorch Metric Learning.Installation$pipinstalltf-simple-metric-learningUsageProvided layers are implemented viatf.keras.layers.LayerAPI, enables;fromtf_simple_metric_learning.layersimportArcFacearcface=ArcFace(num_classes=NUM_CLASSES,margin=MARGIN,scale=SCALE)Example notebook is inexamplesdirectory. Implement CircleLossCL (Class-level label version) layer top of EfficientNet and train it forCars196 dataset;importtensorflowastffromtf_simple_metric_learning.layersimportArcFace,AdaCos,CircleLossCLinputs=tf.keras.layers.Input([*IMAGE_SIZE,3],dtype=tf.uint8)x=tf.cast(inputs,dtype=tf.float32)x=tf.keras.applications.efficientnet.preprocess_input(x)net=tf.keras.applications.EfficientNetB0(include_top=False,weights='imagenet',pooling='avg')embeds=net(x)labels=tf.keras.layers.Input([],dtype=tf.int32)labels_onehot=tf.one_hot(labels,depth=num_classes)# Create metric learning layer# metric_layer = ArcFace(num_classes=num_classes, margin=0.5, scale=64)# metric_layer = AdaCos(num_classes=num_classes)metric_layer=CircleLossCL(num_classes=num_classes,margin=0.25,scale=256)logits=metric_layer([embeds,labels_onehot])model=tf.keras.Model(inputs=[inputs,labels],outputs=logits)model.summary()Note that you should feed labels as inputinto model in training because these layers require labels to forward.In evaluation or prediction, above model requires both images and labels but labels is ignored in those metric learning layers. We only need to use dummy labels (ignored) with the target images because we can't access labels in evaluation or prediction.Referenceshttps://github.com/KevinMusgrave/pytorch-metric-learninghttps://github.com/scikit-learn-contrib/metric-learn
Package: tf-slice-assign
tf-slice-assignA tool for assignment to a slice in TensorFlow.In TensorFlow, as opposed to Pytorch, it is currently impossible to assign to
the slice of a tensor in a range of different settings.
To mitigate this issue,tf-slice-assignintroduces a single function that
allows to do exactly this usingtensor_scatter_nd_update.Usefromtf_slice_assignimportslice_assignnew_tensor=slice_assign(old_tensor,assignment,*slice_args)You can find a relatively simple examplehere.Installationpip install tf-slice-assignList of GitHub issues and StackOverflow questions regarding TensorFlow slice assignmentIn the following table, I am trying to give the reasons as to why no mitigation
for the current problem exists.LinkStatusSOCurrent answer requires creating atf.Variablefor each slice assignment you makeGHQuestion is abouttf.VariableSOAnswers fortf.Variableor usingtensor_scatter_updatein a non-adaptable wayGHSuggestion to usetensor_scatter_nd_updateGHAn answer suggest creating a mask, but a mask can actually be as difficult to create as the indices fortensor_scatter_nd_update
Package: tf-SmeLU
Smooth ReLU in TensorFlowUnofficialTensorFlowreimplementation of the Smooth ReLU (SmeLU) activation function proposed in the
paperReal World Large Scale Recommendation Systems Reproducibility and Smooth Activationsby Gil I. Shamir and Dong Lin.This repository includes an easy-to-use pure TensorFlow implementation of the Smooth ReLU.InstallationThe SmeLU can be installed by usingpip.pipinstallgit+https://github.com/r3v1/tf-SmeLUExample Usageimporttensorflowastffromtf_smeluimportsmelux=tf.range(-6,6,1,dtype=float)# <tf.Tensor: numpy=array([-6., -5., -4., -3., -2., -1.,  0.,  1.,  2.,  3.,  4.,  5.], dtype=float32)>smelu(x,beta=0.1)# <tf.Tensor: numpy=array([0.,0.,0.,0.,0.,0.,0.025,1.,2.,3.,4.,5.], dtype=float32)>smelu(x,beta=0.5)# <tf.Tensor: numpy=array([0.,0.,0.,0.,0.,0.,0.125,1.,2.,3.,4.,5.], dtype=float32)>smelu(x,beta=1.)# <tf.Tensor: numpy=array([0.,0.,0.,0.,0.,0.,0.25 ,1.,2.,3.,4.,5.], dtype=float32)>The SmeLU takes the following parameters.beta: Half-width of a symmetric transition region around x = 0. Defaults to 1.Reference@article{Shamir2022,title={{Real World Large Scale Recommendation Systems Reproducibility and Smooth Activations}},author={Shamir, Gil I and Lin, Dong},journal={{arXiv preprint arXiv:2202.06499}},year={2022}}
Package: tf-som
tf-somTensorflow self-organizing maps.Locally competitive algorithms demonstrate superior convergence to their supervised
counterparts over a suite of tasks. Try them yourself:pipinstalltf-somfrom.modelsimportConvNet# build unsupervised baseunsupervised_base=ConvNet((H,W))# train unsupervisedforx,_intrain_ds:unsupervised_base(x,training=True)...# build supervised headsupervised_head=keras.Sequential([tfkl.Input(unsupervised_base.output_shape),tfkl.Conv2D(8,(3,3),activation='relu'),tfkl.Flatten(),tfkl.Dense(N_classes,activation='softmax')])# assemble full classifierunsupervised_base.trainable=Falseclassifier=keras.Sequential([unsupervised_base,supervised_head])classifier.compile('sgd','cross_entropy','accuracy')# train supervisedclassifier.fit(train_ds)# compare model sizesprint(unsupervised_base.summary())print(supervised_head.summary())