Package: numpycythonpermutations
Efficient NumPy Permutations, Combinations, and Product using Cython/C++/OpenMPGenerate permutations, combinations, and product sets with NumPy efficiently.
The providedgenerate_productfunction is designed to outperform the standard
itertools library, offering more than 20x speed improvement.Utilizes a "Yellow-line-free" Cython Backend for high speed performance.Implements OpenMP multiprocessing for parallel processing.Compiles on the first run (requires a C/C++ compiler installed on your PC).Achieves 90% less memory usage compared to itertools.Performance scales with data size, making it ideal for large datasets.Efficiently creates a lookup NumPy array with a lightweight dtype (typically np.uint8, unless you are combining more than 255 different elements).Utilizes numpy indexing for memory savings - depending on the datatype (and your luck :-) ), numpy shows you only element views, which means, you are saving a loooooooooooooooooooot of memorySupported FunctionalityIteratorArgumentsResultsproduct()p, q, … [repeat=1]cartesian product, equivalent to a nested for-looppermutations()p[, r]r-length tuples, all possible orderings, no repeated elementscombinations()p, rr-length tuples, in sorted order, no repeated elementscombinations_with_replacement()p, rr-length tuples, in sorted order, with repeated elementsGetting StartedOnly tested on Windows 10 / Python 3.11-MakesureyouhavePythonandaC/C++compilerinstalled-UsepipinstallnumpycythonpermutationsordownloaditfromGithubSome examplesGenerating all RGB colors in 200 ms.more than 25 times faster than itertools generating all RGB colorsimportnumpyasnpimportitertoolsfromnumpycythonpermutationsimportgenerate_product# RGB COLORS:args=np.asarray(# The input must be always 2 dimensional (list or numpy)[list(range(256)),list(range(256)),list(range(256)),],dtype=np.uint8,)In[17]:%timeitresus=np.array(list(itertools.product(*args)))5.88s±78.5msperloop(mean±std.dev.of7runs,1loopeach)...:%timeitresus=generate_product(args,remove_duplicates=False,str_format_function=repr,multicpu=True,return_index_only=False,max_reps_rows=-1,r=-1,dummyval="DUMMYVAL")232ms±31.6msperloop(mean±std.dev.of7runs,1loopeach)But even 2.5x faster when using a tiny database2.5xtimesfasterusinglittledataargs=np.asarray([list(range(5)),list(range(5)),list(range(5)),],dtype=np.uint8,)In[23]:%timeitnp.array(list(itertools.product(*args)))39.3µs±113nsperloop(mean±std.dev.of7runs,10,000loopseach)In[25]:%timeitresus=generate_product(args,remove_duplicates=False,str_format_function=repr,multicpu=True,return_index_only=False,max_reps_rows=-1,r=-1,dummyval="DUMMYVAL")19.2µs±176nsperloop(mean±std.dev.of7runs,100,000loopseach)Attention! The output is different (Fortran-styled order) from itertools:Itertoolsarray([[0,0,0],[1,0,0],[2,0,0],...,[253,255,255],[254,255,255],[255,255,255]],dtype=np.uint8)numpycythonpermutationsarray([[0,0,0],[0,0,1],[0,0,2],...,[255,255,253],[255,255,254],[255,255,255]])Deleting duplicatesargs=[[1,2,3,4],[2,0,0,2],[2,1,6,2],[1,2,3,4],]resus1=generate_product(args,remove_duplicates=True,)print(resus1)print(resus1.shape)In[15]:resus1Out[15]:array([[1,2,2,1],[2,0,2,3],[2,2,2,1],[3,2,2,1],...[4,0,1,4],[1,2,6,4],[3,2,6,4],[4,2,6,4],[1,0,6,4],[3,0,6,4],[4,0,6,4]])In[18]:resus1.shapeOut[18]:(96,4)# Without removing duplicatesargs=[[1,2,3,4],[2,0,0,2],[2,1,6,2],[1,2,3,4],]resus2=generate_product(args,remove_duplicates=False,)print(resus2.shape)In[16]:resus2Out[16]:array([[1,2,2,1],[2,2,2,1],[3,2,2,1],...,[2,2,2,4],[3,2,2,4],[4,2,2,4]])In[17]:resus2.shapeOut[17]:(256,4)Filtering DataTo get all colors whose RGB values are R!=G!=BThe order of any filtered output may vary each time due to multicore parsing.args=[list(range(256)),list(range(256)),list(range(256)),]generate_product(args,max_reps_rows=1)array([[119,158,238],[50,2,0],[226,251,90],...,[244,254,255],[245,254,255],[246,254,255]])# But it takes some time to filter 16,7 Million colors:In[38]:%timeitgenerate_product(args,max_reps_rows=1)11.7s±437msperloop(mean±std.dev.of7runs,1loopeach)# Passing a NumPy array is a little fasterargs=np.asarray([list(range(256)),list(range(256)),list(range(256)),],dtype=np.uint8,)In[2]:%timeitgenerate_product(args,max_reps_rows=1)9.94s±209msperloop(mean±std.dev.of7runs,1loopeach)# Another exampleargs=[[2,1,3,4],[4,4,3,4],]resus=generate_product(args,remove_duplicates=True,# removes all duplicated rowsr=len(args[0]),# similar to itertoolsmax_reps_rows=2)# allows only 2 occurrences of the same element in the same row[[11224334][11224343][12213434][21213443][11223344][21123434][21213434][11223434]# Another exampleargs=[[1,2,3,4],]resus=generate_product(args,remove_duplicates=False,r=len(args[0]))print(resus)print(resus.shape)[[1234][2234][3234]...[2123][3123][4123]](256,4)You can mix data typesargs=[[[1,2],3,4],[3,"xxxxx",3,6],[2,0,0,2],[2,0,[0,2]],[8,2,8,2],[4,5,4,5],[[3,3],3,6],[4,5,4,5],[0,{2,3,4},8,7],[1,2,b"xxx3",4],]q=generate_product(args,remove_duplicates=False)Out[6]:array([[list([1,2]),3,2,...,4,0,1],[3,3,2,...,4,0,1],[4,3,2,...,4,0,1],...,[list([1,2]),6,2,...,5,7,4],[3,6,2,...,5,7,4],[4,6,2,...,5,7,4]],dtype=object)thefunctionreprisusuallyusedtofilterNot-Numpy-Friendly-DataThismightleadtosomeproblems,e.g.pandasDataFrameswhichareusuallynotfullyshownwhencalling__repr__Inthesecases,youcanpassacustomfunctiontostr_format_function(buttobehonest:WhothehellputsapandasDataFrameinsideaNumPyarray?)# Example for a function (The string is only used for indexing)str_format_function=(lambdax:x.to_string()ifisinstance(x,pd.DataFrame)elserepr(x))importpandasaspdargs=[[2,1,3,4],[4,4,3,4],[pd.read_csv("https://github.com/datasciencedojo/datasets/blob/master/titanic.csv",on_bad_lines="skip",),np.array([222,3]),dict(baba=333,bibi=444),],]resus=generate_product(args,remove_duplicates=True,r=len(args[0]),max_reps_rows=-1,str_format_function=str_format_function,)print(resus)print(resus.shape)Ain't it pretty? hahaha[[432...{'baba':333,'bibi':444}<!DOCTYPEhtml>0<html1lang="en"2data-color-mode="auto"data-light-theme="lig...3data-a11y-animated-images="system"data-a11y...4>......1062</div>1063<divid="js-global-screen-reader-notice"c...1064<divid="js-global-screen-reader-notice-as...1065</body>1066</html>[1067rowsx1columns]array([222,3])][211...{'baba':333,'bibi':444}{'baba':333,'bibi':444}array([222,3])][113...<!DOCTYPEhtml>0<html1lang="en"2data-color-mode="auto"data-light-theme="lig...3data-a11y-animated-images="system"data-a11y...4>......1062</div>Inhomogeneous Shapes? No problem!# An Inhomogeneous Shape is also no problem.# Just make sure that the default dummy value dummyval="DUMMYVAL" is not in your Array (not very likely, I guess)a=[1,2]b=[3,4]c=[5,6,7]d=[8,9,10]total=[a,b,c,d]resus=generate_product(total,remove_duplicates=True,dummyval="DUMMYVAL")print(resus)[[2369][1358][1369][1468][1458][2358][1479][1379]...[2379][2479][23510][24510][13610][23610][14610][24610][24710]]a=[1,2,3]b=[3,4,4]c=[5,6]d=[8,9,10]total=[a,b,c,d]resus=generate_product(total,remove_duplicates=True,dummyval="DUMMYVAL")print(resus)[[1358][34610][13510][2459][2468][3358]...[3369][23610][2469][14610][3469][33510][13610][14510][24510][34510][33610]]How to get the index# To save memory, the function can only return the index, this saves a lot of memory# and you can access each element by looping through the data and accessing the input Elementargs=[[100,200,300,400],[300,300,300,600],[200,000,000,200],[200,000,000,200],[800,200,800,200],[400,500,400,500],[300,300,300,600],[400,500,400,500],[000,900,800,700],[100,200,300,400],]resus=generate_product(args,remove_duplicates=False,return_index_only=True,)print(resus)print(resus.shape)Thefunctionreturns:[[021...350][121...350][221...350]...[141...793][241...793][341...793]](1048576,10)