Package: reltools
Relation tools for Python.
This relates two data (sorted by certain keys) like SQL joining.Inspired byitertools.groupby,
as long as input data are sorted, almost all processes are evaluated lazily,
which results in the reduction of memory usage.
This feature is for the big data joining without any SQL engines.InstallationInstall withpip.pipinstallreltoolsFeaturesOne-To-ManyOne-to-manyrelationing is provided byrelate_one_to_many.Here, input left-hand-side (lhs) and right-hand-side (rhs)
are sorted in 1st (and also 2nd) keys.>>> lhs = [
...     (1, 'a', 's'),
...     (2, 'a', 't'),
...     (3, 'b', 'u'),
... ]
>>> rhs = [
...     (1, 'a', 'v'),
...     (1, 'b', 'w'),
...     (3, 'b', 'x'),
... ]relate_one_to_manyrelatesrhsitems
to eachlhsitem using the first items as keys.>>> from reltools import relate_one_to_many
>>> one_to_many_related = relate_one_to_many(lhs, rhs)
>>> for left, right in one_to_many_related:
...     left, list(right)
((1, 'a', 's'), [(1, 'a', 'v'), (1, 'b', 'w')])
((2, 'a', 't'), [])
((3, 'b', 'u'), [(3, 'b', 'x')])You can use custom key functions
for not onlyrelate_one_to_manybut also API functions.>>> import operator
>>> custom_key = operator.itemgetter(0, 1)
>>> one_to_many_related = relate_one_to_many(
...     lhs, rhs, lhs_key=custom_key, rhs_key=custom_key)
>>> for left, right in one_to_many_related:
...     left, list(right)
((1, 'a', 's'), [(1, 'a', 'v')])
((2, 'a', 't'), [])
((3, 'b', 'u'), [(3, 'b', 'x')])OneToManyChainerhelps to relate manyrhsiterables tolhs.>>> another_rhs = [
...     ('s', 'f'),
...     ('t', 'g'),
...     ('t', 'h'),
... ]
>>> from reltools import OneToManyChainer
>>> chainer = OneToManyChainer(lhs)
>>> chainer.append(rhs)
>>> chainer.append(
...     another_rhs,
...     lhs_key=operator.itemgetter(2),
...     rhs_key=operator.itemgetter(0),
... )
>>> for left, right, another_right in chainer.chain():
...     left, list(right), list(another_right)
((1, 'a', 's'), [(1, 'a', 'v'), (1, 'b', 'w')], [('s', 'f')])
((2, 'a', 't'), [], [('t', 'g'), ('t', 'h')])
((3, 'b', 'u'), [(3, 'b', 'x')], [])Left Outer JoinLeft outer joining is provided byleft_join.
While SQL left outer joining returns all the combinations,
this returns the pair of items.
Note that therightcan empty, like SQL left joining.>>> from reltools import left_join
>>> lhs = [(1, 'a'), (1, 'b'), (2, 'c'), (4, 'd')]
>>> rhs = [(1, 's'), (1, 't'), (3, 'u'), (4, 'v')]
>>> relations = left_join(lhs, rhs)
>>> for left, right in relations:
...     list(left), list(right)
([(1, 'a'), (1, 'b')], [(1, 's'), (1, 't')])
([(2, 'c')], [])
([(4, 'd')], [(4, 'v')])Right Outer JoinRight outer joining is not supported
because it is left-and-right-opposite of left joining.
Useleft_join(rhs, lhs, rhs_key, lhs_key).Full Outer JoinFull outer joining, which is an original feature ofreltools,
is provided byouter_join.
In contrast toleft_join, full outer joining preserve keys
that are only inrhs.>>> from reltools import outer_join
>>> lhs = [(1, 'a'), (1, 'b'), (2, 'c'), (4, 'd')]
>>> rhs = [(1, 's'), (1, 't'), (3, 'u'), (4, 'v')]
>>> relations = outer_join(lhs, rhs)
>>> for left, right in relations:
...     list(left), list(right)
([(1, 'a'), (1, 'b')], [(1, 's'), (1, 't')])
([(2, 'c')], [])
([], [(3, 'u')])
([(4, 'd')], [(4, 'v')])Inner JoinInner joining is provided byinner_join.
In contrast toleft_join,rightcannot be empty,
like SQL inner joining.>>> from reltools import inner_join
>>> relations = inner_join(lhs, rhs)
>>> for left, right in relations:
...     list(left), list(right)
([(1, 'a'), (1, 'b')], [(1, 's'), (1, 't')])
([(4, 'd')], [(4, 'v')])Many-To-ManySQL-likemany-to-manyrelationing using an internal table is not supported.
This is becausereltoolssupports only sorted data
and does not prefer random accessing.
To achievemany-to-manyrelationing, unnormalize data on preproceing and
use outer joining or inner joining.Memory EfficiencyAlmost all processes are evaluated lazily,
which results in the reduction of memory usage.
(You can try the efficiency by commands likeRELTOOLS_TRY_COUNT=10000000 python3-mdoctest README.rst)>>> import os
>>> n = int(os.environ.get('RELTOOLS_TRY_COUNT', 1000))
>>> lhs = ((i, 'left') for i in range(n))
>>> rhs = ((i, 'right') for i in range(n))
>>> for left, right in relate_one_to_many(lhs, rhs):
...     assert len(list(right)) == 1DevelopmentThis project’s structure is based onPoetry.
All tests are written withdoctestand run withpytest.poetryinstallpoetryrunpytestFor stability, following checks are also run when testing.pyflakespycodestylemypyLicenseCopyright (c) 2018 Yu MOCHIZUKI