END-TO-END TRAINABLE WEAKLY NON-NEGATIVE FACTORIZATION
Takumi Kobayashi and Kenji Watanabe
National Institute of Advanced Industrial Science and Technology
1-1-1 Umezono, Tsukuba, Ibaraki, Japan
ABSTRACT
Non-negative matrix factorization (NMF) is widely applied
to analyze pattern data in an unsupervised manner. It imposes
hard non-negativity constraints on factors to extract intrinsic characteristics from an input matrix, though demanding
complicated optimization techniques which hinder the general applicability. Toward flexible formulation, we propose
weakly non-negative factorization. In contrast to the strict
non-negative approach, our method permits factors to contain small amount of negative values. The relaxation theoretically leads to an efficient factorization formulation which canbe implemented by means of off-the-shelf techniques used ina deep learning literature. Thus, the method is flexibly applicable to versatile factorization tasks, such as deep NMF
and structured NMF. In the experiments on the NMF-related
tasks, we demonstrate that the weak non-negativity produces
effective factors similarly to NMF and the method exhibits favorable performance in comparison to the other approaches.
Index Terms— factorization, weak non-negativity, regularization, leaky ReLU
1. INTRODUCTION
Real-world pattern data, such as images and their feature vectors, are described in a redundant form and thus mining theirintrinsic representation is useful for pattern analysis. It is
plausible to make a low-rank assumption that data are constructed on the basis of only a small number of factors, which
derives diverse factorization methods. While a naive factorization is given by singular value decomposition (SVD) or
PCA, a non-negative matrix factorization (NMF) [1] has beensuccessfully applied in various fields such as signal processing and computer vision [2]. Considering that real-world data
is frequently represented by non-negative physical quantities(e.g., pixel intensities), NMF reveals from pattern matrices
inherent characteristics such as sparsity [1].
NMF is a general framework to optimize two nonnegative factors so that an input matrix is well approximatedby their product. It is formulated as an optimization problem
with non-negativity constraints as shown in the mathematical form (1). Thus, the problem is addressed by various
optimization approaches [2, 3] to cope with the constraints,

and one of the popular techniques is a multiplicative updating [4, 5] which naturally involves non-negativity into
parameter updates. It is dependent on a cost function measuring reconstruction errors; MSE [4] and β-divergence [5]are favorable for the multiplicative updating. However, suchcomplicated optimization techniques and/or dependency onloss functions would degrade general applicability of NMF
such as to a (deep) end-to-end framework [6].
In this paper, we propose a flexible non-negative factorization such that it can be embedded in versatile frameworks
via end-to-end learning. The non-negativity constraint, which
is fundamental for NMF, requires careful treatment by the
optimization technique, making simple gradient descent useless. We formulate weakly non-negative factorization byrelaxing the hard constraint of non-negativity into regularization through reformulation of NMF with re-parameterizedfactors. It permits factors to contain small amount of negative elements while NMF imposes strictly non-negative constraints on the factors. The weakly non-negative factorization
is formulated as an unconstrained problem to which simple
optimization process, such as back-prop gradient descent, iseffectively applicable. It is thus noteworthy that the proposed
method can be implemented by using well-established deeplearning techniques including optimizers which are intensively studied in recent years [7]. Our flexible formulation
is directly applicable to versatile factorization tasks such as
deep NMF [8] and structured NMF.
2. METHOD
We start with a constrained optimization problem for NMF
and then formulate an unconstrained problem to address
weakly non-negative factorization.
2.1. Optimization problem for NMF
Suppose an input matrix X ∈ Rm×n is decomposed into two
factors W ∈ Rm×r and H ∈ Rr×n as X ≈ W H with rankr. By introducing non-negativity constraints into factors [1],we formulate the following optimization problem for NMF;
min `(X, W H),

W ,Hs.t. W ≥ 0, H ≥ 0,(1)

mini-batch samplingwhere an inequality A ≥ 0 indicates element-wise nonnegativity for all components in a matrix A. The loss function` measures discrepancy between X and W H; for NMF, one
can usually employ Euclidean distance or β-divergence toinduce an efficient multiplicative updating [4, 5]. For robust
factorization, L2 -norm regularization with a parameter λ is
imposed on the factors as

MatrixMultiplication

leaky-ReLU

min `(X, W H)+λ(kW k2F +kHk2F ), s.t.W ≥ 0, H ≥ 0.

W ,H

Loss
+ regularization

mini-batch sampling(2)

Fig. 1. Weakly non-negative matrix factorization. It is implemented in an end-to-end mini-batch training with size b.

In order to be further aware of non-negativity, we disentangle
the factors into

where W and H are real-valued matrices and a non-linear
function φη works on a matrix in an element-wise manner as

2.2. Weakly non-negative factorization

W = W+ − W− , H = H+ − H− ,(3)where A+ and A− indicate non-negative matrices responsible for positive and negative parts of a matrix A, respectively.
By using this re-parameterization, similarly to (2), we formulate the following optimization problem;min

W+ ,W− ,H+ ,H−`[X, (W+ − W− )(H+ − H− )](4)+ λ(kW+ k2F + kH+ k2F ) + λ− (kW− k2F + kH− k2F ),s.t. W+ ≥ 0, W− ≥ 0, H+ ≥ 0, H− ≥ 0,

(5)

where we additionally introduce a regularization parameter
λ− for the negative part of factors. It should be noted thatsetting λ− → ∞ enforces W− → 0 and H− → 0 to derive
the original NMF (2). In other words, the parameter λ− controls non-negativity, which thereby motivates us to formulate
weakly non-negative matrix factorization by applying moderately large λ− in comparison to λ. It should be noted that
unlike NMF, in a weakly non-negative matrix, each element
is not strictly non-negative but can take small negative value
if it falls into negative part. The amount of negativity on thefactors is suppressed by the large λ− , especially λ/λ− < 1.
We furtherp
rewrite the negative part matrices W− and H−
by using η = λ/λ− asp
W− = λ/λ− Ŵ− = η Ŵ− , H− = η Ĥ− ,(6)to simplify (4) intomin`[X, (W+ − η Ŵ− )(H+ − η Ĥ− )](7)

W+ ,Ŵ− ,H+ ,Ĥ−+ λ(kW+ k2F + kH+ k2F + kŴ− k2F + kĤ− k2F ),s.t. W+ ≥ 0, Ŵ− ≥ 0, H+ ≥ 0, Ĥ− ≥ 0.

(8)

In this optimization problem, Karush Kuhn–Tucker (KKT)
conditions [9] imply that W+ Ŵ− = 0 and H+ Ĥ− = 0,
meaning that each element either of W+ or Ŵ− is zero. We
can plug this relationship into (7) to provide
min `[X, φη (W )φη (H)] + λ(kW k2F + kHk2F ),

W ,H

(9)


φη (Aij ) =

Aij
ηAij

if Aij ≥ 0= max[Aij , ηAij ], (10)if Aij < 0

where η < 1 (⇔ λ/λ− < 1). The proposed formulation(9, 10) is advantageous in the following two points.
First, the function φη is equivalent to leaky ReLU [10]which is well established andp frequently used in a deep learning literature. Due to η = λ/λ− , the parameter η controls
non-negativity of factors. The original NMF (2) is obtained
simply by setting η = 0 which corresponds to λ− → ∞ in(4). In case of η = 0, however, the non-linear function φη=0
is reduced into ReLU [11], making the problem (9) harder to
optimize. Due to the ReLU function being flat on a negative part, once some elements fall into the negative part, theyhardly enjoy favorable loss gradients. It would accordingly
demand some optimization techniques, such as projected gradients [3], which degrades the flexibility of the formulation(9). Therefore, we simply derive small η  1 from largeλ−  1 for rendering weakly non-negative factorization. In
the another approach, it is conceivable to tune the parameterη during training as in curriculum learning [12]; η is gradually decreased toward 0 so that φη eventually results in ReLU.
These approaches are compared in the experiments of Sec. 3.
Second, the formulation (9) addresses NMF (2) in a wayof unconstrained optimization. Together with the leaky ReLU
φη , it is implemented in a end-to-end framework as shown in
Fig. 1. In contrast to the standard end-to-end method [13],the proposed method does not feed a pattern data, e.g., images, to the model but just samples a subset H (i) ∈ Rr×bfrom the whole factor H ∈ Rr×n in an efficient mini-batchlearning with the mini-batch size b. Besides, off-the-shelfdeep learning techniques are directly applied to optimize it.
For example, we can flexibly employ various types of loss
function `, not limited to MSE and β-divergence, as well as
leverage sophisticated optimizers to minimize the loss; we apply AdamW [7] with a learning rate of 0.001 to an L1 lossof `(x, y) = kx − yk1 . The proposed method is so generalthat the weakly non-negative factorization can be embedded
in various tasks as presented in Sec. 3.

Table 2. Ablation study. Performance score is 100× NMI.

Table 1. Datasets.
ORL face
AR facemulti-feature
ImageNet-CNN# sample# dimension# class

400
2,600
2,000
50,00092 × 112 pixels
120 × 165 × 3 pixels
585 features
2,048 features

40
100
10
1,000

3. EXPERIMENTAL RESULTS
We apply the proposed method to three types of non-negative
factorization; simple NMF (Sec. 3.1), deep NMF (Sec. 3.2)
and deconvolutional factorization (Sec. 3.3).
Dataset. Factorization methods are tested on four datasets(Table 1). ORL face dataset [14] contains 400 images of
92 × 112 gray-scaled pixels captured from 40 subjects. AR
face dataset [15] is constructed by taking 2,600 images of
120 × 165 RGB-color pixels from 100 subjects under various
conditions. Multiple-feature dataset [16] contains 2,000 samples of 585-dimensional non-negative features1 in 10 classes.
We built an ImageNet-CNN dataset by applying pre-trained
ResNet-50 [17] to ImageNet [18] validation set to provide
50,000 samples of 2,048-dimensional feature vectors; notethat ResNet-50 excluding the last FC layer intrinsically produces non-negative features due to ReLU activation.
Evaluation. For quantitatively evaluating the factorization
results, we apply k-means clustering to n latent samples of
H = {hi ∈ Rr }ni=1 and then measure normalized mutual
information (NMI) [19] between the clusters and class labels.
3.1. Non-negative matrix factorization (NMF)
The method (9) is straightforwardly applied to the task of
NMF as X ≈ φη (W )φη (H).
Ablation study. We first analyze the proposed method on
ORL face dataset [14]. As shown in (9), the method is distinctive in terms of the non-linear function φη (x) = max[x, ηx]
and the regularization with λ. We evaluate performance ofthe method in an ablation manner regarding those two aspects in Table 2 where the function φη provides three typesof activation, linear (η = 1), ReLU (η = 0) and leaky-ReLU(η = 0.1), and we set λ = 0.1 to inject L2 regularization.
Without regularization (λ = 0), ReLU activation (η = 0)significantly degrades performance. In this factorization,
the factors W and H are simply initialized by uniform
values including negatives according to the successful training recipe [20]. The ReLU activation provides no updatingwith the initially negative parameters, thereby failing to optimize the factorization. Without regularization, leaky ReLU
(η = 0.1) poorly works in terms of both performance andnon-negativity; Fig. 2 indicates that it fails to produce nonnegative factors. Combining leaky ReLU (η = 0.1) with the
1 We exclude 64 real-value components from whole 649 features.

kφη (x) = max[x, ηx]

20

30

40

50

η

λ

`

67.57
52.17
72.05
76.64
75.71
78.12

75.19
58.76
79.78
83.35
83.36
86.67

80.20
57.71
85.73
88.71
89.08
89.80

81.57
59.82
85.95
87.90
89.25
91.26

1 linear
0 ReLU
0.1 leaky-ReLU
1 linear
0 ReLU
0.1 leaky-ReLU

0
0
0
0.1
0.1
0.1

k · k1k · k1k · k1k · k1k · k1k · k175.24 82.04 86.92 88.29
75.36 79.28 84.27 87.27

10 curriculum
0.1 leaky-ReLU

0.1 k · k1
0.1 k · k22

regularization (λ = 0.1) significantly improves performance.
Even though the method is based on weak non-negativity,
the learnt factors are almost non-negative. We show distribution of weakly non-negative factor components in Fig. 2. Bothfactors, φη (W ) and φη (H), are favorably non-negative; actually, minW ∈W φη (W ) = −0.008 and minH∈H φη (H) =−0.03, meaning that factors are almost non-negative. The basis factor φη (W ) are visualized in Fig. 3a. Similarly to NMF
factors (Fig. 3b), it extracts sparse parts of face appearance
due to the (weak) non-negativity.
The parameter η is set to small η = 0.1 for weakly nonnegative factorization. We also evaluate the curriculum approach (Sec. 2.2) by gradually decreasing η as 1
0 to ensure strict non-negativity of factors in the end. As shown in
Table 2, it is inferior to η = 0.1 and even to η = 0, demonstrating that the constant-η works well with the regularization
without tuning curriculum schedule of η.
The proposed formulation (9) can accept various types ofloss function ` and it is so far set to an L1 loss, `(x, y) = kx−
yk1 . We compare the L1 loss with L2 one which is widely
applied in an NMF framework [1, 4] and Table 2 shows superiority of the L1 loss. Our framework is effectively flexible interms of designing loss functions for improving performance.
Performance comparison. The proposed method is compared with the other NMF methods including the original
NMF [4] that applies multiplicative updating to non-negative
factors, and unfolding NMF [21] which unfolds such multiplicative updating in a deep framework. The performance
results in Table 3a demonstrate the efficacy of our method.
3.2. Deep non-negative matrix factorization
We apply the method in a framework of deep NMF [22]by stacking non-negative factorization in multiple layers.
The proposed method is naturally applicable to stack Lfactorization as, given X ∈ Rm×n ,X ≈ φη (WL ) · · · φη (W1 )φη (H1 ),(11)

where Wl ∈ Rrl+1 ×rl and H1 ∈ Rr1 ×n with rL+1 = m
and l ∈ {1, · · · , L}. The latent representation at the l-thOurs (
w/o reg. (

200K

)
)

Ours (
w/o reg. (

7K

)
)

100K
3K

0

-0.1

0

0.1

0.2

0.3

0.4

0

0.5

-0.5

0

(a) φη (W )

0.5

1.0

1.5

2.0

2.5

(a) Our NMF (92 × 112)(c) Our deconv NMF s = 2 (46 × 56)(b) NMF (92 × 112)(d) Our deconv NMF s = 4 (23 × 28)

3.0(b) φη (H)

Fig. 2. Histograms of factor elements.

Fig. 3. Learnt factors H whose resolutions are shown in parentheses.

Table 3. Performance comparison on tasks of NMF (a) and deep NMF (b). For deep NMF, two-layer ranks (r1 , r2 ) are set to(20, 40) on ORL, (50, 100) on AR, (10, 20) on multi-feature and (500, 1000) on ImageNet-CNN.
ORL
30

AR
40

50

50

75

100

150

2

multi-feature
5
10

20

500

ImageNet-CNN
750 1000 1500(a) NMF

20

NMF
67.12 75.36 82.51 84.79 33.08 39.37 43.19 48.59 29.21 43.73 65.78 63.72 75.63 78.89 80.12 80.57
Unfold [21] 66.03 70.46 73.84 74.54 33.17 37.34 40.81 44.74 26.64 51.23 61.95 63.91 57.63 59.38 60.76 62.82
Ours (9)78.12 86.67 89.80 91.26 32.88 39.43 44.31 52.36 37.61 68.75 85.51 76.15 77.11 80.05 81.01 81.16(b) deep NMFkNMF l = 1
NMF l = 2
Ours l = 1
Ours l = 2

74.19
71.09
74.45
74.79

77.99
78.89
82.59
80.01

82.06
79.63
84.20
83.74

84.82
84.00
86.11
85.67

32.10
30.32
30.64
31.98

38.83
37.42
39.18
37.33

43.14
41.63
42.84
41.58

Table 4. Deconvolutional factorization on ORL dataset.
(a) Framework(b) Performance results

stride

30

40

50

raw
deconv20

s = 2 74.60 81.53 85.39 86.93
s = 4 71.75 79.73 83.31 83.87

sumks = 2 76.40 84.34 88.98 89.31
s = 4 79.08 82.72 87.12 89.50

layer is given by φη (Wl−1 ) · · · φη (W1 )φη (H1 ); the l = 1st layer provides simply φη (H1 ) as latent. While there aresome variants of deep NMF incorporating additional techniques [8, 23, 24], for fair comparison, the proposed method
is compared with a deep NMF which updates factors in a multiplicative form with L2 regularization of factors. In this experiment, we apply L = 2-layered NMF and report performance in Table 3b; two ranks (r1 , r2 ) on respective datasets
are shown in the caption. Our method favorably works on
decomposition of an input matrix into multiple factors.
3.3. Deconvolutional factorization

33.95
30.90
37.39
36.62

55.98
53.86
66.85
67.23

62.73
64.94
84.11
78.60

64.59
66.41
76.99
76.37

35.51
35.83
76.84
76.93

39.60
39.71
79.88
79.85

42.23
42.37
80.94
81.16

45.68
45.75
81.03
81.25

where ∗Ts indicates a deconvolutional operator with step sizes which means upscaling by a factor of s, the i-th image isdenoted by X (i) ∈ Rh×w , and two factors of rank r are givenhw
as W ∈ R s × s ×r and H (i) ∈ R2s×2s×r . This factorization
contrasts with [26, 27] which address convolutional decomposition of a single temporal sequence without upscaling. The
proposed deconvolutional factorization is evaluated on ORL
face dataset to report performance in Table 4 as well as show
factors in Fig. 3cd. The raw latent space is straightforwardly
given by flattening H (i) into 4s2 r-dimensional feature vector.
The factor H (i) , however, contains detailed upscaling filter
weights which are less relevant to latent representation. Thus,
for more favorable latent space, we measure significance offactors at respective ranks by marginalizing out spatial filterP(i)elements { x,y Hxyr0 }rr0 =1 to produce r-dimensional latentvectors akin to the standard NMF latent ones (Sec. 3.1); theapproach is denoted by sum in Table 4. The factors (Fig. 3d)
are ×16 smaller than those of NMF (Fig. 3a) while producing
favorable performance in comparison to Table 3a.
4. CONCLUSION

Finally, we leverage our flexible method to factorize images
in a deconvolutional manner. A standard NMF (Sec. 3.1) explores factor images W of the same resolution as input images. For further efficient representation, we extract smallersized factor images which can reconstruct an input image bymeans of deconvolution [25] equipped with step (upscaling)
size s and kernel filters of 2s×2s size to be optimized as well.
The deconvolutional factorization is described as
X (i) ≈ φη (W ) ∗Ts φη (H (i) ),

49.22
48.13
49.72
47.97

(12)

We have proposed a weakly non-negative factorization method.
The method is theoretically derived from an NMF optimization formulation by relaxing non-negativity constraints into
regularization via re-parameterization of factors. It efficiently
provides almost non-negative factors by means of off-theshelf deep learning techniques in an end-to-end fashion. The
experimental results demonstrate that the proposed method is
flexibly applicable to versatile NMF-related frameworks andproduces favorable performance on those tasks.

5. REFERENCES[1] Daniel D. Lee and Hyunjune S. Seung, “Learning theparts of objects by non-negative matrix factorization,”
Nature, vol. 401, pp. 788–791, 1999.
[2] Yu-Xiong Wang and Yu-Jin Zhang, “Nonnegative matrix factorization: A comprehensive review,” IEEE
Transactions on Knowledge and Data Engineering, vol.
25, no. 6, pp. 1336–1353, 2013.

[15] Aleix M. Martı́nez and Avinash C. Kak, “Pca versuslda,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 23,
no. 2, pp. 228–233, 2001.
[16] Dheeru Dua and Casey Graff,“UCImachinelearning
repository,”
2017,
http://archive.ics.uci.edu/ml/datasets/Multiple+Features.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian
Sun, “Deep residual learning for image recognition,” in
CVPR, 2016, pp. 770–778.

[3] Naiyang Guan, Dacheng Tao, Zhigang Luo, andBo Yuan, “Nenmf: An optimal gradient method for nonnegative matrix factorization,” IEEE Transactions on
Signal Processing, vol. 60, no. 6, pp. 2882–2898, 2012.

[18] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li,
and Li Fei-Fei, “Imagenet: A large-scale hierarchicalimage database,” in CVPR, 2009, pp. 248–255.

[4] Daniel D. Lee and Hyunjune S. Seung, “Algorithms for
non-negative matrix factorization,” in NeurIPS, 2000,
pp. 556–562.

[19] Haifeng Liu, Zhaohui Wu, Xuelong Li, Deng Cai, andThomas S. Huang, “Constrained nonnegative matrix
factorization for image representation,” IEEE Trans.
Pattern Anal. Mach. Intell., vol. 34, no. 7, pp. 1299–
1311, 2012.

[5] Cédric Févotte and Jérôme Idier, “Algorithms for nonnegative matrix factorization with the β-divergence,”
Neural Comput., vol. 23, no. 9, pp. 2421–2456, 2011.
[6] Ian Goodfellow, Yoshua Bengio, and Aaron Courville,
Deep Learning, MIT Press, 2016.
[7] Ilya Loshchilov and Frank Hutter, “Decoupled weightdecay regularization,” in ICLR, 2019.
[8] Pierre De Handschutter, Nicolas Gillis, and Xavier
Siebert, “A survey on deep matrix factorizations,” Computer Science Review, vol. 42, pp. 100423, 2021.
[9] Stephen Boyd and Lieven Vandenberghe, Convex Optimization, Cambridge University Press, 2004.
[10] Andrew L. Maas, Awni Y. Hannun, and Andrew Y. Ng,“Rectifier nonlinearities improve neural network acoustic models,” in ICML, 2013.
[11] Vinod Nair and Geoffrey E. Hinton, “Rectified linear
units improve restricted boltzmann machines,” in ICML,
2010, pp. 807–814.
[12] Yoshua Bengio dn Jerome Louradour, Ronan Collobert,
and Jason Weston, “Curriculum learning,” in ICML,
2009.
[13] Paris Smaragdis and Shrikant Venkataramani, “A neuralnetwork alternative to non-negative audio models,” in
ICASSP, 2017, pp. 86–90.
[14] F.S. Samaria and A.C. Harter, “Parameterisation of astochastic model for human face identification,” in IEEE
Workshop on Applications of Computer Vision, 1994,
pp. 138–142.

[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian
Sun, “Delving deep into rectifiers: Surpassing humanlevel performance on imagenet classification,” in ICCV,2015.
[21] Rami Nasser, Yonina C. Eldar, and Roded Sharan,“Deep unfolding for non-negative matrix factorization
with application to mutational signature analysis,” J.
Comput. Biol., vol. 29, no. 1, pp. 45–55, 2022.
[22] Andrzej Cichocki and Rafal Zdunek, “Multilayer nonnegative matrix factorisation,” Electronics Letters, vol.
42, no. 16, pp. 947–948, 2006.
[23] Jianyong Sun, Qingming Kong, and Zongben Xu,“Deep alternating non-negative matrix factorisation,”
Knowledge-Based Systems, vol. 251, pp. 109210, 2022.
[24] Pierre De Handschutter, Nicolas Gillis, and Xavier
Siebert, “Hierarchical feature extraction by multi-layer
non-negative matrix factorization network for classification task,” Neurocomputing, vol. 165, pp. 63–74, 2015.
[25] Matthew D. Zeiler, Dilip Krishnan, Graham W. Taylor,
and Rob Fergus, “Deconvolutional networks,” in CVPR,
2010.
[26] Paris Smaragdis, “Non-negative matrix factor deconvolution; extraction of multiple sound sources from monophonic inputs,” in ICA, 2004, pp. 494–499.
[27] Mikkel N. Schmidt and Morten Mørup, “Non-negative
matrix factor deconvolution; extraction of multiple
sound sources from monophonic inputs,” in ICA, 2006,
pp. 700–707.