Package: pycut-pursuit
Cut-Pursuit Algorithms, Parallelized Along ComponentsGeneric C++ classes for implementing cut-pursuit algorithms.Specialization to convex problems involvinggraph total variation, and nonconvex problems involvingcontour length, as explained in our articles(Landrieu and Obozinski, 2016; Raguet and Landrieu, 2018).Parallel implementation with OpenMP.MEX interfaces for GNU Octave or Matlab.Extension modules for Python.This Git repository uses submodules.Clone withgit clone --recurse-submodules https://gitlab.com/1a7r0ch3/parallel-cut-pursuitPull changes withgit pull --recurse-submodulesTable of ContentsGeneral problem statementC++ classes and Specializations2.1.Proximity operator of the graph total variation2.2.Quadratic functional and graph total variation2.3.Separable multidimensional loss and graph total variation2.4.Separable distance and contour lengthDocumentation3.1.Directory tree3.2.Graph structure3.3.C++ documentation3.4.GNU Octave or Matlab3.5.PythonReferencesLicenseGeneral problem statementThe cut-pursuit algorithms minimize functionals structured, over a weighted graphG= (V,E,w), asF:x∈ ΩV↦f(x) +
∑(u,v) ∈Ew(u,v)ψ(xu,xv) ,where Ω is some base set, and the functionalψ: Ω² → ℝ penalizes dissimilarity between its arguments, in order to enforce solutions which arepiecewise constant along the graphG.The cut-pursuit approach is to seek partitionsVof the set of verticesV, constituting the constant connected components of the solution, by successively solving the corresponding problem, structured over the reduced graphG= (V,E), that isarg minξ∈ ΩVF(x) ,    such that ∀U∈V, ∀u∈U,xu=ξU,and then refining the partition.A key requirement is thus the ability to solve the reduced problem, which often have the exact same structure as the original one, but with much less vertices |V| ≪ |V|. If the solution of the original problem has only few constant connected components in comparison to the number of vertices, the cut-pursuit strategy can speed-up minimization by several orders of magnitude.Cut-pursuit algorithms come in two main flavors, namely “directionally differentiable” and “noncontinuous”.In thedirectionally differentiablecase, the base set Ω is typically a vector space, and it is required thatfis differentiable, or at least that its nondifferentiable part isseparable along the graphand admits (potentially infinite)directional derivatives. This comprises notably many convex problems, whereψ(xu,xv) = ║xu−xv║, that is to say involving agraph total variation. The refinement of the partition is based on the search for a steep directional derivative, and the reduced problem is solved using convex or continuous optimization; optimality guarantees can be provided.In thenoncontinuouscase, the dissimilarity penalization typically usesψ(xu,xv) = 0 ifxu=xv, 1 otherwise, resulting in a measure of thecontour lengthof the constant connected components. The functionalfis typically required to be separable along the graph, and to have computational properties favorable enough for solving reduced problems. The refinement of the partition relies on greedy heuristics.Both flavors admit multidimensional extensions, that is to say Ω is not required to be only a set of scalars.C++ classes and SpecializationsThe modulemaxflowimplements the classMaxflow, a modification of theGraphclass of Y. Boykov and V. Kolmogorov, for making use of theirmaximum flow algorithm.The modulecut_pursuitimplements the base classCp, defining all steps of the cut-pursuit approach in virtual methods.The modulecut_pursuit_d1implements the classCp_d1derived fromCp, specializing cut-pursuit for directionally differentiable cases involving the graph total variation.The modulecut_pursuit_d1implements the classCp_d0derived fromCp, specializing cut-pursuit for noncontinuous cases involving the contour length penalization.Cp_prox_tv: proximity operator of the graph total variationAlso coined “graph total variation denoising” or “general fused LASSO signal approximation”. The objective functional isF:x∈ ℝD⨯V↦
1/2 ║y−x║Mℓ2+
∑(u,v) ∈Ew(u,v)║xu−xv║p,Mδ,whereDis the dimension of the signal on each vertex,y∈ ℝD⨯V,Mℓis a diagonal metric weighting the square ℓ2norm,w∈ ℝEare regularization weights,
and the norm on the finite differences is defined bypbeing 1 or 2 and a weighting diagonal metricMδ.The reduced problem is solved using thepreconditioned forward-Douglas–Rachford splitting algorithm, included as a git submodulepcd-prox-split.Cp_d1_ql1b: quadratic functional, ℓ1norm, bounds, and graph total variationThe base set is Ω = ℝ, and the general form isF:x∈ ℝV↦
1/2 ║y(ℓ2)−Ax║2+
∑v∈Vλv|y(ℓ1)−xv| +∑v∈Vι[mv,Mv](xv) +
∑(u,v) ∈Ew(u,v)|xu−xv| ,wherey(ℓ2)∈ ℝn,A: ℝV→ ℝnis a linear operator,y(ℓ1)∈ ℝVandλ∈ ℝVandw∈ ℝEare regularization weights,m,M∈ ℝVare parameters andι[a,b]is the convex indicator of [a,b] : x ↦ 0 ifx∈ [a,b], +∞ otherwise.Wheny(ℓ1)is zero, the combination of ℓ1norm and total variation is sometimes coinedfused LASSO.WhenAis the identity,λis zero and there are no box constraints, the problem boils down tothe proximity operator of the graph total variation.Currently,Amust be provided as a matrix. See the documentation for special cases.The reduced problem is solved using thepreconditioned forward-Douglas–Rachford splitting algorithm, included as a git submodulepcd-prox-split.An example withGNU Octave or MatlabandPythoninterfaces, whereAis a full ill-conditioned matrix, with positivity and fused LASSO constraints, on a task ofbrain source identification from electroencephalography.ground truthraw retrieved activityidentified sourcesCp_d1_lsx: separable loss, simplex constraints, and graph total variationThe base set is Ω = ℝD, whereDcan be seen as a set of labels, and the general form isF:x∈ ℝD⨯V↦f(y,x) +
∑v∈VιΔD(xv) +∑(u,v) ∈Ew(d1)(u,v)∑d∈Dλd|xu,d−xv,d| ,wherey∈ ℝD⨯V,fis a loss functional (see below),w(d1)∈ ℝEandλ∈ ℝDare regularization weights, andιΔDis the convex indicator of the simplex
ΔD= {x∈ ℝD| ∑dxd= 1 and ∀d,xd≥ 0}:x↦ 0 ifx∈ ΔD, +∞ otherwise.The following loss functionals are available, wherew(f)∈ ℝVare weights on vertices.Linear:f(y,x) = − ∑v∈Vw(f)v∑d∈Dxv,dyv,dQuadratic:f(y,x) = ∑v∈Vw(f)v∑d∈D(xv,d−yv,d)2Smoothed Kullback–Leibler divergence (equivalent to cross-entropy):f(y,x) = ∑v∈Vw(f)vKL(αu+ (1 −α)yv,αu+ (1 −α)xv),whereα∈ ]0,1[,u∈ ΔDis the uniform discrete distribution overD,
and
KL: (p,q) ↦ ∑d∈Dpdlog(pd/qd).The reduced problem is solved using thepreconditioned forward-Douglas–Rachford splitting algorithm, included as a git submodulepcd-prox-split.An example with the smoothed Kullback–Leibler is provided withGNU Octave or MatlabandPythoninterfaces, on a task ofspatial regularization of semantic classification of a 3D point cloud.ground truthrandom forest classifierregularized classificationCp_d0_dist: separable distance and weighted contour lengthThe base set is Ω = ℝDor ΔDand the general form isF:x∈ ℝD⨯V↦f(y,x) +
∑(u,v) ∈Ew(d0)(u,v)║xu−xv║0,wherey∈ ΩV,fis a loss functional akin to a distance (see below), and
║·║0is the ℓ0pseudo-normx↦ 0 ifx= 0, 1 otherwise.The following  loss functionals are available, wherew(f)∈ ℝVare weights on vertices andm(f)∈ ℝDare weights on coordinates.Weighted quadratic: Ω = ℝDandf(y,x) = ∑v∈Vw(f)v∑d∈Dm(f)d(xv,d−yv,d)2Weighted smoothed Kullback–Leibler divergence (equivalent to cross-entropy):
Ω = ΔDandf(y,x) = ∑v∈Vw(f)vKLm(f)(αu+ (1 −α)yv,αu+ (1 −α)xv),whereα∈ ]0,1[,u∈ ΔDis the uniform discrete distribution overD,
andKLm(f): (p,q) ↦ ∑d∈Dm(f)dpdlog(pd/qd).The reduced problem amounts to averaging, and the split step usesk-means++ algorithm.When the loss is quadratic, the resulting problem is sometimes coined “minimal partition problem”.An example with the smoothed Kullback–Leibler is provided withGNU Octave or Matlabinterface, on a task ofspatial regularization of semantic classification of a 3D point cloud.DocumentationDirectory tree.   
├── include/        C++ headers, with some doc  
├── octave/         GNU Octave or Matlab code  
│   ├── doc/        some documentation  
│   └── mex/        MEX C++ interfaces
├── pcd-prox-split/ git submodule preconditionned forward-Douglas–Rachford 
│                   algorithm (required only for directionnaly 
│                   differentiable cases and example data)
├── python/         Python code  
│   ├── cpython/    C Python interfaces  
│   └── wrappers/   python wrappers and documentation  
├── src/            C++ sources  
└── wth-element/    git submodule for weighted quantiles search
                    (required only for cp_d1_ql1b)C++ documentationRequiresC++11.Be sure to have OpenMP enabled with your compiler to enjoy parallelization. Note that, as of 2020, MSVC still does not support OpenMP 3.0 (published in 2008); consider switching to a decent compiler.The number of parallel threads used in parallel regions is crucial for good performance; it is roughly controlled by a preprocessor macroMIN_OPS_PER_THREADwhich can be again set with-Dcompilation flag. A rule of thumb is to set it to10000on personal computers with a handful of cores, and up to100000for large computer clusters with tens of cores.The C++ classes are documented within the corresponding headers ininclude/.Graph structureGraph structures must be given as forward-star representation. For conversion from simple adjacency list representation, or for creation from scratch for regular N-dimensionnal grids (2D for images, 3D for volumes, etc.), see thepcd-prox-split/grid-graphgit submodule.GNU Octave or MatlabSee the scriptcompile_parallel_cut_pursuit_mex.mfor typical compilation commands; it can be run directly from the GNU Octave interpreter, but Matlab users must set compilation flags directly on the command lineCXXFLAGS = ...andLDFLAGS = ....The integer type holding the components assignment is by defaut on 16 bits. For applications expecting a large number of components, this can be extended to 32 bits with the compilation option-DCOMP_T_ON_32_BITS.Extensive documentation of the MEX interfaces can be found within dedicated.mfiles inoctave/doc/.The scriptexample_prox_tv.mexemplifies the use ofCp_prox_tv, on a task ofcolor image denoising.The scriptexample_EEG.mexemplifies the use ofCp_d1_ql1b, on a task ofbrain source identification from electroencephalography.The scriptsexample_labeling_3D.mandexample_labeling_3D_d0.mexemplify the use of, respectively,Cp_d1_lsxandCp_d0_dist, on a task ofspatial regularization of semantic classification of a 3D point cloud.PythonRequiresnumpypackage.See the scriptsetup.pyfor compiling modules withsetuptools; it can be run simply by usingpipe.g.python -m pip install .. pre compiled binaries for Windows and Linux will soon be available onPyPI.if more than 65535 components are expected in you graph you can force the use of 32 bit indices by setting
theCOMP_T_ON_32_BITSenvironement variable to 1 e.g.export COMP_T_ON_32_BITS=1on bash.Extensive documentation of the Python wrappers can be found in the corresponding.pyfiles.The scriptexample_prox_tv.pyexemplifies the use ofCp_prox_tv, on a task ofcolor image denoising.The scriptexample_EEG.pyexemplifies the use ofCp_d1_ql1b, on a task ofbrain source identification from electroencephalography.The scriptsexample_labeling_3D.pyandexample_labeling_3D_d0.pyexemplify the use of, respectively,Cp_d1_lsxandCp_d0_dist, on a task ofspatial regularization of semantic classification of a 3D point cloud.ReferencesL. Landrieu and G. Obozinski,Cut Pursuit: Fast Algorithms to Learn Piecewise Constant Functions on Weighted Graphs, 2017.H. Raguet and L. Landrieu,Cut-pursuit Algorithm for Regularizing Nonsmooth Functionals with Graph Total Variation, 2018.Y. Boykov and V. Kolmogorov, An Experimental Comparison of Min-Cut/Max-Flow Algorithms for Energy Minimization in Vision, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2004.LicenseThis software is under the GPLv3 license.