Package: tensors-data-class
TensorsDataClassPyTorch Extension Library for organizing tensors in a form of a structured tree of dataclasses, with built-in support for advanced collating mechanisms. The batch creation process seamlessly solves issues like: sequences padding, un/flattening variable #objects per example into a single batch dimension, fixing within-example indices to be batch-based indices, auto-creation of sequences & collate masks, and more.What pains TensorsDataClass aims to solve... variable number of sequences per example where the sequence lengths may also be variable; lots of inputs usually gets messy - hard to handle, to name, to move to GPU, to abstract in a (X,Y) fashion ...Installationpipinstalltensors-data-classUsage example# TODO: simplify the below example. still use these:#   BatchFlattenedSeq, BatchFlattenedTensor,#   BatchedFlattenedIndicesFlattenedTensor,#   BatchedFlattenedIndicesFlattenedSeq,#   BatchedFlattenedIndicesPseudoRandomPermutationBatchedFlattenedIndicesPseudoRandomPermutation,#   BatchFlattenedPseudoRandomSamplerFromRangefromtensors_data_classimport*@dataclasses.dataclassclassCodeExpressionTokensSequenceInputTensors(TensorsDataClass):token_type:BatchFlattenedSeq# (nr_expressions_in_batch, batch_max_nr_tokens_in_expr)kos_token_index:BatchFlattenedTensor# (nr_kos_tokens_in_all_expressions_in_batch,)identifier_index:BatchedFlattenedIndicesFlattenedTensor# (nr_identifier_tokens_in_all_expressions_in_batch,)@dataclasses.dataclassclassSymbolsInputTensors(TensorsDataClass):symbols_identifier_indices:BatchedFlattenedIndicesFlattenedTensor# (nr_symbols_in_batch,);  value meaning: identifier batched indexsymbols_appearances_symbol_idx:BatchedFlattenedIndicesFlattenedTensor# (nr_symbols_appearances,);symbols_appearances_expression_token_idx:BatchFlattenedTensor=None# (nr_symbols_appearances,);symbols_appearances_cfg_expression_idx:BatchedFlattenedIndicesFlattenedTensor=None# (nr_symbols_appearances,);@dataclasses.dataclassclassCFGPathsInputTensors(TensorsDataClass):nodes_indices:BatchedFlattenedIndicesFlattenedSeqedges_types:BatchFlattenedSeq@dataclasses.dataclassclassCFGPathsNGramsInputTensors(TensorsDataClass):nodes_indices:BatchedFlattenedIndicesFlattenedSeqedges_types:BatchFlattenedSeq@dataclasses.dataclassclassPDGInputTensors(TensorsDataClass):cfg_nodes_control_kind:Optional[BatchFlattenedTensor]=None# (nr_cfg_nodes_in_batch, )cfg_nodes_has_expression_mask:Optional[BatchFlattenedTensor]=None# (nr_cfg_nodes_in_batch, )cfg_nodes_tokenized_expressions:Optional[CodeExpressionTokensSequenceInputTensors]=Nonecfg_nodes_random_permutation:Optional[BatchedFlattenedIndicesPseudoRandomPermutation]=Nonecfg_control_flow_paths:Optional[CFGPathsInputTensors]=Nonecfg_control_flow_paths_ngrams:Optional[Dict[int,CFGPathsNGramsInputTensors]]=None@dataclasses.dataclassclassIdentifiersInputTensors(TensorsDataClass):sub_parts_batch:BatchFlattenedTensor# (nr_sub_parts_in_batch, )identifier_sub_parts_index:BatchedFlattenedIndicesFlattenedSeq# (nr_identifiers_in_batch, batch_max_nr_sub_parts_in_identifier)identifier_sub_parts_vocab_word_index:BatchFlattenedSeq# (nr_identifiers_in_batch, batch_max_nr_sub_parts_in_identifier)identifier_sub_parts_hashings:BatchFlattenedSeq# (nr_identifiers_in_batch, batch_max_nr_sub_parts_in_identifier, nr_hashing_features)sub_parts_obfuscation:BatchFlattenedPseudoRandomSamplerFromRange# (nr_sub_parts_obfuscation_embeddings)@dataclasses.dataclassclassMethodCodeInputTensors(TensorsDataClass):example_hash:stridentifiers:IdentifiersInputTensorssymbols:SymbolsInputTensorsmethod_tokenized_code:Optional[CodeExpressionTokensSequenceInputTensors]=Nonepdg:Optional[PDGInputTensors]=Noneexample1=MethodCodeInputTensors(...)# TODO: fill example dataexample2=MethodCodeInputTensors(...)# TODO: fill example databatch=MethodCodeInputTensors.collate([example1,example2])print(batch)# TODO: add example for creating a padded-sequence (after applying embedding on the input), unflattening.Different types for different use-casesTensorsDataClassBatchFlattenedTensorBatchFlattenedSeqBatchedFlattenedIndicesFlattenedTensorBatchedFlattenedIndicesFlattenedSeqBatchedFlattenedIndicesPseudoRandomPermutationBatchedFlattenedIndicesPseudoRandomPermutationBatchFlattenedPseudoRandomSamplerFromRange