Package: torchgel
torch-gelThis package provides PyTorch implementations to solve the group elastic net
problem. LetAj(j = 1 … p) be feature matrices of sizesm ×
nj(mis the number of samples, andnjis the number
of features in thejthgroup), and letybe anm × 1vector of
the responses. Group elastic net finds coefficientsβj, and a biasβ0that solve the optimization problemminβ0, …, βp½ ║y - β0- ∑ Ajβj║2+m ∑ √nj(λ1║βj║+ λ2║βj║2).Hereλ1andλ2are scalar coefficients that control
the amount of 2-norm and squared 2-norm regularization. This 2-norm
regularization encourages sparsity at the group level; entireβjmight become 0. The squared 2-norm regularization is in similar spirit to
elastic net, and addresses some of the issues of lasso. Note that group elastic
net includes as special cases group lasso (λ2= 0), ridge
regression (λ1= 0), elastic net (eachnj= 1), and
lasso (eachnj= 1andλ2= 0). The optimization
problem is convex, and can be solved efficiently. This package provides two
implementations; one based on proximal gradient descent, and one based on
coordinate descent.InstallationInstall withpippipinstalltorchgeltqdm(for progress bars), and numpy are pulled in as dependencies. PyTorch
(v1.0+) is also needed, and needs to be installed manually. Refer to thePyTorch websitefor instructions.Usageexamples/main.ipynbis a Jupyter notebook that walks
through using the package for a typical use-case. A more formal description of
the functions follows; and for details about the algorithms, refer to the
docstrings of files in thegeldirectory.Solving Single InstancesThe modulesgel.gelfistaandgel.gelcdprovide implementations based on
proximal gradient descent and coordinate descent respectively. Both have similar
interfaces, and expose two main public functions:make_Aandgel_solve. The
feature matrices should be stored in a list (sayAs) as PyTorch tensor
matrices, and the responses should be stored in a PyTorch vector (sayy).
Additionally, the sizes of the groups (nj) should be stored in a
vector (sayns). First use themake_Afunction to convert the feature
matrices into a suitable format:A=make_A(As,ns)Then passA,yand other required arguments togel_solve. The general
interface is::b_0,B=gel_solve(A,y,l_1,l_2,ns,**kwargs)l_1andl_2are floats representingλ1andλ2respectively. The method returns a floatb_0representing the bias and a
PyTorch matrixBholding the other coefficients.Bhas sizep ×maxjnjwith suitable zero padding. The following
sections cover additional details for the specific implementations.Proximal Gradient Descent (FISTA)Thegel.gelfistamodule contains a proximal gradient descent implementation.
It's usage is just as described in the template above. Refer to the docstring
forgel.gelfista.gel_solvefor details about the other arguments.Coordinate DescentThegel.gelcdmodule contains a coordinate descent implementation. Its usage
is a bit more involved than the FISTA implementation. Coordinate descent
iteratively solves single blocks (each corresponding to a singleβj). There are multiple solvers provided to solve the individual
blocks. These are thegel.gelcd.block_solve_*functions. Refer to their
docstrings for details about their arguments.gel.gelcd.gel_solverequires
passing a block solve function and its arguments (as a dictionary). Refer to
its docstring for further details.Solution Pathsgel.gelpathsprovides a wrapper functiongel_pathsto solve the group
elastic net problem for multiple values of the regularization coefficients. It
implements a two-stage process. For a givenλ1andλ2,
first the group elastic net problem is solved and the feature blocks with
non-zero coefficients is extracted (the support). Then ridge regression models
are learned for each of several provided regularization values. The final model
is summarized using an arbitrary provided summary function, and the summary for
each combination of the regularization values is returned as a dictionary. The
docstring contains more details.gel.ridgepathscontains another useful function,ridge_pathswhich can efficiently solve ridge regression for multiple
regularization values.CitationIf you find this code useful in your research, please cite@misc{koushik2017torchgel,
  author = {Koushik, Jayanth},
  title = {torch-gel},
  year = {2017},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/jayanthkoushik/torch-gel}},
}