Package: carling
CarlingViaWikipedia:Carlings are pieces of timber laid fore and aft under the deck of a ship, from one beam to another.
They serve as a foundation for the whole body of the ship.Useful transforms for supporting our apache beam pipelines.Mapping transform utilscarling.LabelLabels all elements.carling.SelectRemoves all columns which are not specified in*keys.carling.ProjectTransforms each element into a tuple of values of the specified columns.carling.IndexByTransforms each elementVinto a tuple(K, V).Kis the projection ofVby*keys, which is equal to the tuple
produced by theProjecttransform.carling.StringifyTransforms each element into its JSON representation.carling.IndexBySingleTransforms each elementVinto a tuple(K, V).The difference betweenIndexBySingle(key)andIndexBy(key)with a single
key is as follows:IndexBySingleproduces the index as a plain value.IndexByproduces the index as a single-element tuple.carling.RenameFromToRename columns according tofrom_to_key_mapping.carling.ExcludeRemoves all columns specified in*keys.Grouping transform utilsGeneric grouping transform utilscarling.UniqueOnlyProduces elements that are the only elements per key after deduplication.Given aPCollectionof(K, V),
this transform produces the collection of allVs that do not share
the same correspondingKs with any other elements after deduplicating
all equivalent(K, V)pairs.This transform is equivalent toSingletonOnlywithapache_beam.Distinct.[(1, "A"), (2, "B1"), (2, "B2"), (3, "C"), (3, "C"), (4, "A")]will be
transformed into["A", "C", "A"].carling.SingletonOnlyProduces elements that are the only elements per key.Given aPCollectionof(K, V),
this transform produces the collection of allVs that do not share
the same correspondingKs with any other elements.[(1, "A"), (2, "B1"), (2, "B2"), (3, "C"), (3, "C"), (4, "A")]will be
transformed into["A", "A"].carling.IntersectionProduces the intersection of givenPCollections.Given a list ofPCollections,
this transform produces every element that appears in all collections of
the list.
Elements are deduplicated before taking the intersection.carling.FilterByKeyFilters elements by their keys.The constructor receives one or morePCollections ofKs,
which are regarded as key lists.
Given aPCollectionof(K, V),
this transform discards all elements withKs that do not appear
in the key lists.If multiple collections are given to the constructor,
this transform treats the intersection of them as the key list.carling.FilterByKeyUsingSideInputFilters a single collection by a single lookup collection, using a common key.Given: - aPCollection(lookup_entries) of(V), as a lookup collection - aPCollection(pcoll) of(V), as values to be filtered - a common key (filter_key)A dictionary calledfilter_dict- is created by mapping the value offilter_keyfor each entry inlookup_entriesto True.Then, for each item in pcoll, the value associated withfilter_keycheckd againstfilter_dict, and if it is found, the entry passes through. Otherwise, the entry is
discarded.Note:lookup_entrieswill be used as aside input, so care
must be taken regarding the size of thelookup_entriescarling.DifferencePerKeyProduces the difference per key between twoPCollections.Given twoPCollections ofV,
this transform indexes the collections by the specified keysprimary_keys,
compares corresponding twoVlists for everyK,
and produces the difference perK.
If there is no difference, this transform produces nothing.TwoVlists are considered to be different if the numbers of elements
differ or two elements of the lists with a same index differ
at one of the specified columnscolumns.carling.SortedSelectPerKeyGroups items by a set ofkeys-- column names per rowEmits the "MAX"valuefor each collection as defined by thekey_fnCan emit "MIN" by passingreverse=Truekwargcarling.PartitionRowsContainingNoneEmits two tagged PCollections:Default (result[None]): Rows are guaranteed not to have anyNonevaluesresult["contains_none"]: Rows for which at least one column had aNonevalueCategoricalcarling.CreateCategoricalDictsFor a set of columnular data inputs this function takes:- cat_cols:

    Type: `[str]`

    An array of "categorical" columns

- existing_dicts:

    Type: `PCollection[(string, string, int)]`

    Rows of tuples of type:
    (column, previously_seen_value, mapped_unique_int)

    Mapping a set of "previously seen" keys to unique int values for each
    column.
    Not optional.
    If none exist, pass an empty PCollection.It then creates a transform which takes a pcollection and- looks at the input pcoll for unseen values in each categorical column
- creates new unique integers for each distinct unseen value, starting at max(previous value for column)+1
- ammends the existing mappings with (col, unseen_value, new_unique_int)Output is:- Type: `PCollection[(string, string, int)]`This is useful for preparing data to be trained by eg. LightGBMcarling.ReplaceCategoricalColumnsUtilizes the "categorical dictionary rows" generated byCreateCategoricalDictswhich is a list of pairs of type of(column, value,unique_int).Replaces each column with the appropriate value found in the mapping.Test Utilscarling.test_utils.pprint_equal_toThis module contains theequal_tofunction from apache beam, but adapted to output results using pretty print. Reading the results as a large, unformatted string makes it harder to pick out what changed/is missing.General Utilcarling.LogSamplePrint items of the givenPCollectionto the log.LogSampleprints the JSON representations of the input items to the Python's
standard logging system.To avoid too much log entries being printed,LogSamplelimits the number of
logged items. The constructor parameterndetermines the limit.By default,LogSampleprints logs with theINFOlog level. The constructor
parameterleveldetermines the level.carling.ReifyMultiValueOptionPrepares multi-value delimited options. Useful in contexts where
you want to create a multi-value option in a template environment.inputs:delimited string optionoptional delimiter string (default is "|")output:Type:PCollection[str]