Package: tfmc-machine-parts-pp
tmfc-machine-parts-ppplanted partitions over machine part structuresThis python module offers data and functionality to generate planted partitions over directed acyclic graphs with dissimilarities. The directed acyclic graphs represent machine parts, where the vertices are parts, and the edges are "part-of" relations. The planted partitions simulate a copy-paste scenario where a machine (graph) is duplicated, but the parts (vertices) are given new identifiers, and the copy-relation is forgotten. The planted partitions represent the copy-paste links, and the presented problem is to recover the planted partition from the family of graphs and the dissimilarities.The machine parts data is extracted from a large database of subsea oil and gas machinery owned by TechnipFMC (https://www.technipfmc.com/), and the dissimilarities are obtained from an in-company data science project using machine learning to develop a similarity measure over the machine parts calculated from metadata.For a detailed description of the data and the planted partition generation mechanism, please see [LINK TO DOC].Requirementspython 3+numpy##License
The software in this module is licensed under the GNU Lesser General Public License (https://www.gnu.org/licenses/lgpl-3.0.en.html).Installationpython -m pip install tfmc-machine-parts-pp [--user]How to get hold of the dataThe data can be obtained either as filesparts.csvdissimilarities.csvOr they can be obtained through the python API as follows:importtfmc.machine_parts_ppasppparts=pp.get_all_machine_parts()dists=pp.get_all_dissimilarities()In the above code,partsis adictmapping a part id (integer) to a list of part-ids, namely to the list of directly contained parts. That is;bis apart ofaifb in parts[a].
The objectdistsis a symmetricN x Nnumpyarray holding the pairwise dissimilarities of all the machine parts.However, if the goal is to use the data to generate clustering problems, the following sections give a description of how the APIs can be used for this.Examples involving planted partitionsThe below examples demonstrate how toObtain a generated problem instance together with a planted partition.Solve the problem using a library for clustering. We provide two examples, one usingdbscan, and one example usingophac.Evaluate the quality of the solution by computing the adjusted Rand index relative the planted partition. We employ the python libraryclusimfor this computation.Obtaining a problem instance with a planted partitionWe start by obtaining the problem instance and the planted partitionimporttfmc.machine_parts_ppaspp# number of copy-paste operationsmult=5# The base data before copy-pastingbase=pp.get_all_machine_parts()# mean and variance of the noise to be added to the dissimilarity datamean=0.05var=0.15# Generate the problem and obtain the planted partitiongraph,dissims,planted_partition=pp.planted_partition(base,mult,mean,var)We now havegraph- a dict on the form (x,[y1,y2,...])wherexis an integer and[y1,...]are the "parts of"x, i.e. the descendants ofxwhen we considergraphto be a directed acyclic graph.dissims- AnNxNnumpyarray with pair-wise dissimilarities of the vertices in graph. All dissimilarities are in the range[0,1].planted_partition- A list of lists, each list a cluster, and each cluster a collection of copy-paste pairs.We can now use this for testing and benchmarking of algorithms that take(graph,dissims)as input and try to use this information to recoverplanted_partition.The following two examples continue by first clustering the data, and then evaluate the result by computing the adjusted rand index.Using DBSCAN for clusteringThis example requires the python modulessklernandclusimto be installed.The full source can be downloaded via this link:example_dbscan.pyfromsklearn.clusterimportDBSCAN# Use dbscan to cluster, using 3 as the core point threshold (min_samples),# and specifying metric='precomputed', since we have pre-computed dissimilaritiesdbscan=DBSCAN(eps=0.085,min_samples=3,metric='precomputed')clustering=dbscan.fit(dissims)# number of clustersnClusters=max(*clustering.labels_)+1# extracting the physical clustersnElts=len(graph)clusters=[[]for_inrange(nClusters)]foreltinrange(nElts):clusters[clustering.labels_[elt]].append(elt)# We can now compare the 'clustering' clusters with the 'planted_partition'# clusters to evaluate the degree of precision in recovering the planted# partition## We have chosen to use the library 'clusim', which requires the clustering to# be represented by a dict mapping (eltId,clustId)# See the documentaton of clusim for details.importclusim.clusteringimportclusim.sim# Planted partition clustering representationpp_cluster_idx=pp.clustering_to_cluster_index_mapping(planted_partition)pp_dict={x:[pp_cluster_idx[x]]forxinrange(nElts)}pp_clst=clusim.clustering.Clustering(elm2clu_dict=pp_dict)# dbscan clustering representationdb_dict={x:[clustering.labels_[x]]forxinrange(nElts)}db_clst=clusim.clustering.Clustering(elm2clu_dict=db_dict)# Since the dbscan clustering is not pre-set to have a fixed number of clusters,# we consider the clustering to be "drawn" from the family of all clusteringsari=clusim.sim.adjrand_index(db_clst,pp_clst,'all1')print('Adjusted Rand index:',ari)Using ophac for clusteringophacis a library that is tailored especially for clustering of directed acyclic graphs (seehttps://bitbucket.org/Bakkelund/ophac/wiki/Homefor details).This example requires the python modulesophacandclusimto be installed.The full source can be downloaded via this link:example_ophac.py# We now cluster the elements using order preserving clustering through the# ophac library. See the documentation of ophac for details.importophac.hierarchyashcimportophac.dtypesasdt# Formatting the part-of relations as required by ophacN=len(graph)Q=[sorted(graph[x])forxinrange(N)]# Formatting the dissimilarity as required by ophacD=[]foriinrange(N):forjinrange(i+1,N):D.append(dissims[i,j])# Run the ophac approximation algorithmac=hc.approx_linkage(D,Q,'complete')# Plotting the merge dissimilarities to find the location of the# so called knee. Inspection of the below plot shows that a useful# cutoff may be at about 150 merges --> we extract the clustering# at this pointimportmatplotlib.pyplotaspltnCut=150plt.plot(ac[0].dists,label='merges')plt.plot([nCut],[ac[0].dists[nCut]],c='red',marker='o',label=('cut at%dmerges'%nCut))plt.legend()plt.title('Choosing to cut at%dmerges based on inspection'%nCut)plt.show()# Using ophac libraries to extract the clustering at nCut mergescutoff_partition=dt.merge(dt.Partition(n=N),ac[0].joins[:nCut])clusters=cutoff_partition.data# We can now compare the 'clusters' with the 'planted_partition'# to evaluate the degree of precision in recovering the planted partition.## We have chosen to use the library 'clusim', which requires the clustering to# be represented by a dict mapping (eltId,clustId)# See the documentaton of clusim for details.importclusim.clusteringimportclusim.sim# Planted partition clusim representationpp_cluster_idx=pp.clustering_to_cluster_index_mapping(planted_partition)pp_dict={x:[pp_cluster_idx[x]]forxinrange(N)}pp_clst=clusim.clustering.Clustering(elm2clu_dict=pp_dict)# ophac clustering clusim representationdb_cluster_idx=pp.clustering_to_cluster_index_mapping(clusters)db_dict={x:[db_cluster_idx[x]]forxinrange(N)}db_clst=clusim.clustering.Clustering(elm2clu_dict=db_dict)# Since the ophac clustering is not pre-set to have a fixed number of clusters,# we consider the clustering to be "drawn" from the family of all clusteringsari=clusim.sim.adjrand_index(db_clst,pp_clst,'all1')print('Adjusted Rand index:',ari)