Package: dskit
DSKitDSKit (Data Science Kit) is a Python package that provides tools for solving simple Data Science routine problems.InstallingpipinstalldskitTutorialDSKit consists of two submodules:dskit.frame- contains a set of functions forpandas.DataFrameandpandas.Seriesmanipulation.dskit.tensor- contains a set of functions fornumpy.ndarraymanipulation.dskit.framedummifierdummifieris less harmful alternative topd.get_dummies. This function takes aDict[str, Tuple[object, ...]]and returns aCallable[[pd.DataFrame], pd.DataFrame]which takes a frame and returns a dummified frame. Key of the dictionary is treated as a name of a column and value of the dictionary is treated as a set of unique values of that column.dummifieralso takes an optional parameternamewhich has a typeCallable[[str, object], str]. Thenamefunction takes a name of a column and a unique value of that column to produce a name of a column in a dummified frame. The default implementation of thenameis:lambda n, x: n + "_" + str(x).dummifierusesencoderfunction under the hood.xs=pd.DataFrame({"A":(1,2,2,5,5),"B":("a","a","b","c","d")})dummify=dummifier(dict(xs))print(dummify(xs))#    A_1  A_2  A_5  B_a  B_b  B_c  B_d# 0  1.0  0.0  0.0  1.0  0.0  0.0  0.0# 1  0.0  1.0  0.0  1.0  0.0  0.0  0.0# 2  0.0  1.0  0.0  0.0  1.0  0.0  0.0# 3  0.0  0.0  1.0  0.0  0.0  1.0  0.0# 4  0.0  0.0  1.0  0.0  0.0  0.0  1.0ys=pd.DataFrame({"C":(True,True,False,True),"A":(1,2,3,4)})print(dummify(ys))#        C  A_1  A_2  A_5# 0   True  1.0  0.0  0.0# 1   True  0.0  1.0  0.0# 2  False  0.0  0.0  0.0# 3   True  0.0  0.0  0.0One of the reasons whydummifieris less harmful thanpd.get_dummiesis that it will not dummify new values. Thanks to that Machine Learning models will operate on data with the same number of dimensions regardless of new values presence in a new portion of data.old_frame=pd.DataFrame({"B":("a","a","b")})dummify=dummifier(dict(old_frame))new_frame=pd.DataFrame({"B":("a","b","c")})print(dummify(new_frame))#    B_a  B_b# 0  1.0  0.0# 1  0.0  1.0# 2  0.0  0.0print(pd.get_dummies(new_frame))#    B_a  B_b  B_c# 0    1    0    0# 1    0    1    0# 2    0    0    1encoderencoderis a function which takes a set of values and returns aCallable[[Tuple[object, ...]], pd.DataFrame]. The returned function one-hot-encodes passed values.encoderalso takes an optional parameternamewhich has a typeCallable[[object], str]. Thenamefunction takes a unique value from the passed set to produce a name of a column in a one-hot-encoded frame. The default implementation of thenameis:str. This function usessklearn.preprocessing.OneHotEncoderunder the hood.encoded=encoder((1,2,3))((1,2,3,4,np.nan))print(encoded)#      1    2    3# 0  1.0  0.0  0.0# 1  0.0  1.0  0.0# 2  0.0  0.0  1.0# 3  0.0  0.0  0.0# 4  0.0  0.0  0.0encoded=encoder((1,2,3),name=lambdax:"column_"+str(x))((1,2,3,4,np.nan))print(encoded)#    column_1  column_2  column_3# 0       1.0       0.0       0.0# 1       0.0       1.0       0.0# 2       0.0       0.0       1.0# 3       0.0       0.0       0.0# 4       0.0       0.0       0.0dskit.tensorbatchbatchis a function which takes aTuple[Tuple[np.ndarray, ...], ...], transposes it and appliesnp.stackon each element resulting in aTuple[np.ndarray, ...].xs=((np.array([1,2,3]),np.array([4,5]),np.ones((2,3))),(np.array([7,8,9]),np.array([5,4]),np.zeros((2,3))))x,y,z=batch(xs)print(x)print("="*5)print(y)print("="*5)print(z)# [[1 2 3]#  [7 8 9]]# =====# [[4 5]#  [5 4]]# =====# [[[1. 1. 1.]#   [1. 1. 1.]]##  [[0. 0. 0.]#   [0. 0. 0.]]]batchesbatchesis a function which takes a sliding window lengthnand astep, and returns a function which takes anIterable[Tuple[np.ndarray, ...]], applies sliding window over it and usesbatchfunction on each window. This function returns anIterable[Tuple[np.ndarray, ...]]. Each window has length equal ton. In case whenexact=Falseoption is passed, each window has at most length equal ton.stepis simply a shift of a sliding window. By defaultstepis equal ton.xs=np.arange(15).reshape(-1,3)ys=np.arange(10).reshape(-1,2)print(xs)# [[ 0  1  2]#  [ 3  4  5]#  [ 6  7  8]#  [ 9 10 11]#  [12 13 14]]print(ys)# [[0 1]#  [2 3]#  [4 5]#  [6 7]#  [8 9]]forx,yinbatches(n=3)(zip(xs,ys)):print(x)print("="*5)print(y)print()# [[0 1 2]#  [3 4 5]#  [6 7 8]]# =====# [[0 1]#  [2 3]#  [4 5]]#forx,yinbatches(n=3,step=2,exact=False)(zip(xs,ys)):print(x)print("="*5)print(y)print()# [[0 1 2]#  [3 4 5]#  [6 7 8]]# =====# [[0 1]#  [2 3]#  [4 5]]## [[ 6  7  8]#  [ 9 10 11]#  [12 13 14]]# =====# [[4 5]#  [6 7]#  [8 9]]## [[12 13 14]]# =====# [[8 9]]#cyclecycleis a multidimensional version ofitertools.cyclefunction. This function takes anp.ndarraywithTuple[int, ...]and returns "cycled"np.ndarray.xs=np.arange(4).reshape(-1,2)print(xs)# [[0 1]#  [2 3]]cycled_xs=cycle(xs,(3,3))print(cycled_xs)# [[0 1 0 1 0 1]#  [2 3 2 3 2 3]#  [0 1 0 1 0 1]#  [2 3 2 3 2 3]#  [0 1 0 1 0 1]#  [2 3 2 3 2 3]]zeros=cycle(0,(2,2,3))print(zeros)# [[[0 0 0]#   [0 0 0]]##  [[0 0 0]#   [0 0 0]]]gridrangegridrangeis a function similar to Python'srangefunction. The difference betweengridrangeandrangeis thatgridrangeoperates onTuple[int, ...]instead ofint.forxingridrange((2,3)):print(x)# (0, 0)# (0, 1)# (0, 2)# (1, 0)# (1, 1)# (1, 2)forxingridrange((1,1),(3,4)):print(x)# (1, 1)# (1, 2)# (1, 3)# (2, 1)# (2, 2)# (2, 3)forxingridrange((1,1),(10,20),(5,5)):print(x)# (1, 1)# (1, 6)# (1, 11)# (1, 16)# (6, 1)# (6, 6)# (6, 11)# (6, 16)iteraxisiteraxisis a function which takes anp.ndarrayand returnsIterable[np.ndarray]along passed axis. This function is similar tonp.apply_along_axis. The difference betweeniteraxisandnp.apply_along_axisis thatnp.apply_along_axisapplies some function to arrays, wheniteraxisreturns those arrays.xs=np.arange(27).reshape(-1,3,3)forxiniteraxis(xs,axis=-1):print(x)# [0 1 2]# [3 4 5]# [6 7 8]# [ 9 10 11]# [12 13 14]# [15 16 17]# [18 19 20]# [21 22 23]# [24 25 26]movemoveallows you to move sourcenp.ndarrayto destinationnp.ndarrayat coordinateTuple[int, ...].moveworks on a copy of thedestinationarray unlessinplace=Trueis passed. The default coordinate is(0, 0, ...).xs=np.arange(4).reshape(-1,2)ys=np.zeros((3,3),dtype=np.uint)moved=move(xs,ys,coordinate=(1,1))print(moved)# [[0 0 0]#  [0 0 1]#  [0 2 3]]xs=np.arange(4).reshape(-1,2)ys=np.zeros((3,3),dtype=np.uint)_=move(xs,ys,inplace=True)print(ys)# [[0 1 0]#  [2 3 0]#  [0 0 0]]slicesslicesis simply:RawSlice=Union[Tuple[Optional[int]],Tuple[Optional[int],Optional[int]],Tuple[Optional[int],Optional[int],Optional[int]]]defslices(xs:Iterable[RawSlice])->Tuple[slice,...]:returntuple(starmap(slice,xs))Example ofslicesusage:xs=np.arange(9).reshape(-1,3)ys=(1,None),(0,1)print(xs[slices(ys)])# [[3]#  [6]]# same asprint(xs[1:,0:1])# [[3]#  [6]