High-resolution Hyperspectral Imaging via Matrix Factorization
Rei Kawakami∗
The University of Tokyo

John Wright
Microsoft Research Asia

Yu-Wing Tai
KAIST

rei@cvl.iis.u-tokyo.ac.jp

jowrig@microsoft.com

yuwing@cs.kaist.ac.kr

Yasuyuki Matsushita
Microsoft Research Asia

Moshe Ben-Ezra
Microsoft Research Asia

Katsushi Ikeuchi
The University of Tokyo

yasumat@microsoft.com

mosheb@microsoft.com

ki@cvl.iis.u-tokyo.ac.jp

Abstract
Hyperspectral imaging is a promising tool for applications in geosensing, cultural heritage and beyond. However,compared to current RGB cameras, existing hyperspectral
cameras are severely limited in spatial resolution. In thispaper, we introduce a simple new technique for reconstructing a very high-resolution hyperspectral image from tworeadily obtained measurements: A lower-resolution hyperspectral image and a high-resolution RGB image. Our approach is divided into two stages: We first apply an unmixing algorithm to the hyperspectral input, to estimate a basis
representing reflectance spectra. We then use this representation in conjunction with the RGB input to produce thedesired result. Our approach to unmixing is motivated by
the spatial sparsity of the hyperspectral input, and casts theunmixing problem as the search for a factorization of the input into a basis and a set of maximally sparse coefficients.
Experiments show that this simple approach performs reasonably well on both simulations and real data examples.

1. Introduction
The spectrum of scene radiance is characterized by a
distribution of intensity at each wavelength. Conventional
RGB cameras perform a very gross quantization of this
distribution, integrating the product of the intensity withthree basis functions that are significantly spread over wavelength. Inevitably, information about the scene radiance,and hence about the physical constituents of the scene, islost in this process.
The goal of hyperspectral imaging is to acquire much
more faithful representations of the scene radiance. The
typical output of a hyperspectral imaging system is a setof intensity values that represent the integrals of the radiance against many basis functions, each of which is welllocalized in wavelength. The availability of such a detailed,physical representation of the scene is critical for some applications. For example, in medical image analysis, full in∗ This work was done while the first author was visiting Microsoft Research Asia.

formation about the scene radiance can be used to detect
anomalies that are not distinguishable from RGB images
alone. Also, faithful color reproduction is critical for museums to record their artwork. Moreover, the use of hyperspectral imagery has been shown to enhance the performance of a number of computer vision tasks, including segmentation, tracking, and recognition [18].
Unfortunately, this utility comes at a price: the spatial resolution of current hyperspectral imaging systems isseverely limited compared to that of RGB cameras. Recording images with a number of spectral bands requires not
only a larger number of exposures but also longer exposure
times to ensure good signal-to-noise ratio in low-light conditions. It is therefore very interesting to investigate newimaging mechanisms which can acquire detailed radianceinformation without sacrificing spatial resolution.
Contributions: In this paper, we develop a practical highresolution hyperspectral imaging system based on a highresolution RGB camera and low-resolution hyperspectral
camera. We introduce a two-step process for combining
these two measurements to estimate a high-resolution hyperspectral image. First, using a principled solution to thespectral unmixing problem [13], we obtain an optimal setof basis functions for representing the reflectances of thevarious materials in the scene. Motivated by the simple observation that there are likely to only be a few materials
contributing to each pixel in the hyperspectral image, wecast the unmixing problem as one of sparse matrix factorization, and solve it via `1 -minimization. We use the basis obtained from this method, together with the high-resolution
RGB observation to reconstruct the spectrum at each location. The proposed method can efficiently obtain very accurate spectrum estimates with a significant resolution gap between the high-resolution RGB camera and low-resolution
hyperspectral camera.

2. Related work
The spectrum can be densely measured using diffraction
gratings, prisms, color filters or changing illumination. Du

et al. used a prism and a mask to capture spectral images
at video rate [8]. Color filters with a black/white camera
are also widely used. The liquid crystal tunable filter [9]is popular since it has narrow passband that can be electronically adjusted. Schechner et al. [28] used a linear variable filter that has varying passband in one filter dimension.
A custom-made linear CCD with changing color filters areused in [24]. A spectral video camera using two sets of
CCD with different spectral filters are proposed in [32].
Illumination based methods modulate the light sources
so that they emit lights of a specific wavelength. Zuzak et
al. [34] used broadband light diffracted with digital micromirror device (DMD) to select which wavelength to be
on or off. Park et al. [20] used five types of LEDs and an
RGB camera to capture multi-spectral video.
It is not trivial to increase the spatial resolution of thosecameras, and some of them are limited in the spectral resolution. Higher resolution in sensor itself would not solve theproblem at once, since it can further cut down the number ofphotons reaching to the sensors, though the high-resolution
in the spectral bandwidth already reduces them.
Motivated by the difficulty of building higher resolution
hyperspectral sensors, researchers in remote sensing andcolor research have proposed a number of methods to fuse ahigh-resolution panchromatic image with a low-resolution
hyperspectral image [6, 12, 2]. One popular technique first
applies a linear transformation of color coordinates (e.g.
HSI) to separate chromaticity and luminance into different channels. Since our human eyes are more sensitive tothe luminance, this technique replace the luminance by the
one from the high-resolution panchromatic image and thenreverse the linear transformation to reconstruct the highresolution multi-spectral images. These methods, however,tend to either distort the spectrum or blur the result since thevalues in chromaticity channels are not upsampled.
Another class of methods use image filtering techniques
to interpolate the spectra in higher resolution [14]. Since
these methods assume the interpolated values are spatiallysmooth, they often over-smooth the image details in the reconstructed hyper-/multi-spectral images. In contrast, ourmethod respects the relationship between the RGB andhyperspectral channels, ensuring that the recovered highresolution hyperspectral image produces the same highresolution RGB image as the input. It also ensures thatthe more detailed spectral properties of the recovered image agree with those of the hyperspectral input.
Our approach accomplishes this by drawing on tools developed in the sparse representation community [4]. Conceptually, these tools are a good fit for our problem, since
their goal sense signals and images using small (seemingly
insufficient) sets of linear measurements. This seemingly
impossible task is made possible by introducing strong priorinformation, in the form of the assumption that the signal issparse in a given basis [25]. Naturally, to apply these tech-niques, we need to know that the signal is indeed sparseor compressible in a known basis, such as a wavelet basis.
As we will see, in our problem, due to the special structure of the measurements, we may assume a much stronger
form of sparsity: each hyperspectral pixel images only a few
materials. However, this strong prior information comeswith a price: unlike most applications of compressed sensing, here, the basis of sparsity, which corresponds to the reflectances of the materials in the scene is not known aheadof time, and must be estimated from the observation itself(similar to [15]).
Recently, a number of groups have applied similar techniques from sparse signal representation to hyperspectral
data problems such as supervised classification [5] and hyperspectral unmixing [35, 17]. Our approach also makes
use of hyperspectral unmixing as a key step, and uses analgorithm motivated by the sparsity of the signal in the spatial dimension. Our approach to unmixing has technicaldifferences from the aforementioned works. For example,
whereas [17] assumes sparsity in both the spectral and spatial domains, we only work with an assumption of spatial
sparsity that follows naturally from the physical setting of
the problem. However, the most important difference between the aforementioned works and our approach is that
we utilized an inhomogeneous set of measurements: hyperspectral and RGB, and these different measurements havevery different spatial sampling rates.
In the context of this paper, the most relevant related
work is [23], which investigates the utility of wavelet sparsity priors for hyperspectral acquisition. Similar to our setting, this paper also augments the compressive hyperspectral measurements with an RGB image. However, as we willsee, this work differs from ours in several respects: first, it
requires a special hardware setup to obtain compressive hyperspectral measurements. Second, obtaining compressive
measurements that multiplex over space may obscure a very
natural spatial sparsity that arises due to the presence of alimited number of materials in the scene. We anticipate thatfor scenes in which the materials vary slowly relative to thesampling rate of the hyperspectral image, our method will
be able to obtain a more accurate reconstruction by directly
exploiting this sparsity.

3. Problem formulation
The goal of this work is to acquire a high-resolution
spectral signal Z ∈ RW ×H×S , where W , H and S are theimage width, the image height, and the number of sampled
wavelengths respectively. Our system takes as input two
types of measurements: An RGB image with a high spatial resolution Yrgb ∈ RW ×H×3 , and a lower-resolution
hyperspectral image Yhs ∈ Rw×h×S , where w  W andh  H. Both of these inputs are linear functions of thetarget signal:
Yhs = Phs [Z], Yrgb = Prgb [Z],(1)

where Phs : RW ×H×S → Rw×h×S and Prgb :RW ×H×S → RW ×H×3 are linear mappings.1
Since in general the total number of observations obtained from the high-resolution RGB and low-resolution
hyperspectral images is smaller than unknowns whS +3W H  W HS, the problem is highly under-constrained reconstructing Z is impossible without further assumptions.
The first is that the number M of distinct materials in thescene is relatively small. Let rj ∈ RS denote the reflectancefunction of the j-th material. If the illumination is uniform,
then the radiance aj of the j-th material will be the sameat all points containing that material. Hence, at any point(i, j) in space, the spectrum Z(i, j, ∗) ∈ RS satisfies
M
X

Z(i, j, ∗) ≈.
am hm (i, j) = Ah(i, j),

(2)

m=1

where am ∈ RS is the m-th radiance corresponds
to the reflectance of the m-th material, and h(i, j) =[h1 (i, j), . . . , hM (i, j)] are the coefficients.
Now, each observation Yhs (i, j) can be written asX
Yhs (i, j, ∗) ∝Z(l, k, ∗)k,l∈Wij= AX.
h(k, l) = Aq(i, j), (3)min kQk1 s.t. Ỹhs = AQ, kAei k2 = 1 ∀ i.
A,Q

where Wij is an appropriate window of the high-resolution
image, and q(i, j) ∈ RM are the summed coefficients. Second, this method assumes that at each pixel (i, j) of the
low-resolution Yhs , only a few of the materials are present.
Hence, the vector q(i, j) is a sparse vector - only a few of its
entries are nonzero. This assumption is valid whenever thematerials in the scene are not changing too quickly relative
to the spatial sampling frequency of Yhs .
1

4. Solution via ` -minimization
If we concatenate the equations (3) for all pixels (i, j),this yields a matrix equation(4).
where Ỹhs = [Yhs (1, 1, ∗), . . . , Yhs (w, h, ∗)] ∈ RS×wh ,.
and Q = [q(1, 1), . . . , q(w, h)] ∈ RM ×wh . Eq. (4) expresses the hyperspectral observations as a product of two
physically meaningful terms: The basis A of reflectancefunctions, and the sparse coefficients Q which represent thefractions of each material present at each location. These
two matrices are initially unknown. However, for scenesthat obey our above assumptions, we do know that the matrix Q should be sparse; ideally, as sparse as possible.
1 Concretely, an RGB camera multiplies sensitivity functions f (λ)
cto the incoming radiane z(λ) and integrates over the spectral range Ω:RΩ fc (λ)z(λ)dλ, where c = {r, g, b}. In the matrix form, Yrgb = FZ,
where F is a 3 × S sensitivity matrix and Z is a S × W H matrix.

(5)

This approach has been suggested and analyzed, e.g., in[11].3 We solve Eq. (5) via a non-smooth Gauss-Newton algorithm that repeatedly linearizes the nonlinear constraints
and solves a sequence of `1 -norm minimization problems.
As a more detailed description is beyond the scope of thispaper, we refer the interested reader to [10].
Once the basis A is recovered, we reconstruct Z using sparsity as a guide. Assuming A has been correctly
estimated, each pixel of the target high-resolution image
should admit a sparse representation in terms of the columnsof A: Z(i, j, ∗) = Ah(i, j). We seek the sparsest coefficients h(i, j) that satisfy the measurement equation
Yrgb (i, j, ∗) = Prgb Ah(i, j):h(i, j) =s.t.

k,l∈WijỸhs = AQ,This is fortuitous: Although there are many possible decompositions of Ỹhs as a product AQ, under fairly mild
circumstances the sparsest such decomposition is unique.2
Physically, such a decomposition attempts to explain eachobservation in Yhs using the fewest number of materials
possible. We seek such a decomposition by locally minimizing the `1 norm (sum of magnitudes) of the coefficientssubject to a data constraint and a constraint on the `2 normof the columns of A:

argminh khk1
kYrgb (i, j, ∗) − Prgb Ahk2 ≤ ε.

(6)Above, ε is a parameter that reflects the noise level in the
RGB image. We can then use the estimated coefficients ĥto form our estimate of Z:
Ẑ(i, j, ∗) = Aĥ(i, j).

(7)

5. Experiments
To assess the performance of the proposed method, we
conduct both simulation and real-world experiments. We
compare the proposed method with the component substitution method [2] as well as three methods based on Principal Component Analysis (PCA) [22] that we have developed. In Section 5.2, we use these methods as baselines
for a quantitative comparison with real hyperspectral data
and a simulated RGB image. Finally, in Sec. 5.3, we show
the effectiveness of our method in practice, using images
acquired with our own hardware setup.
2 For example, if n ≤ S and Q has Bernoulli nonzeros with magnitudes
following from any absolutely continuous probability distribution, it is notdifficult to show that with high probability the decomposition is uniqueprovided wh = Ω(n log n).
3 The reader familiar with the sparse approximation literature may recognize the problem of factoring Ỹhs into A and Q as an instance of the“dictionary learning” problem discussed in the survey paper [26] and references therein. Study of this problem in imaging dates back at least to
the seminal work of Olshausen and Field [19], since which many local optimization algorithms have been proposed. We expect that many of thesealgorithms, such as the K-SVD [1], can also be profitably applied to our
problem.

Balloons

Beads

SpongesOil painting

Flowers

CD

Figure 1. Simulated RGB images using the database [33].

MethodBalloons

Beads

Sponges

CSM [2]GlobalLocal window
RGB clustering
Factorization

13.9
6.9/4.7
7.0/4.9
6.6/4.3
3.0/3.0

28.5
10.5/8.8
10.6/8.9
9.7/7.9
9.2/9.2

19.9
15.4/12.3
14.0/10.6
13.6/10.0
3.7/3.7

Image set
Oil painting Flowers
12.2
5.4/3.8
5.7/4.1
5.5/4.0
4.7 / 4.7

14.4
9.8/8.9
7.5/6.3
7.8/6.5
5.4/5.4

CD
13.3
10.3/10.0
9.6/9.2
9.1/8.6
8.2/8.2

Fake and real
peppers

Fake and real
peppers
13.7
7.1/5.9
8.8/8.0
8.5/7.6
4.7/4.7

Photo and facePhoto
and face
13.1
4.7/3.8
10.9/10.5
4.7/3.8
3.3/3.3

Table 1. Numerical comparison of RMSE (Root Mean Squared Error) between methods. Two numbers are before/after the back-projection
refinement. The numbers are shown in the range of 8-bit images. Those in bold are the best score among the methods.

5.1. Baselines: PCA Variants
The approaches in this section are motivated by the empirical observation that real scene radiances vary smoothlywith wavelength, and hence can be represented using relatively small numbers of basis functions [16, 30, 7, 21].
One natural approach to obtaining basis is to apply Principal Component Analysis (PCA) to the hyperspectral observations.
Let APCA ∈ RS×3 be a matrix whose columns are thefirst three principal components of Ȳ. If the columns of
APCA capture most of the variance in the hyperspectral image, and they are not orthogonal to Prgb , we may be able touse them to estimate Z, via
q(i, j) = (Prgb APCA )† Yrgb (i, j, ∗),ẐPCA (i, j, ∗) = APCA q(i, j),
†

(8)
(9)

where (·) denotes the matrix pseudo-inverse. We call thisapproach the “Global PCA” method.
Since the variation of spectrum across the whole scene
can be larger than three dimensions, the straightforward
global method might not accurately reconstruct the hyperspectral image. In this case, it may be better to replace
the global linear model of PCA by a local linear model,
given by fitting data in a neighborhood of each given pixel.
Because these local windows overlap, the estimated local
bases share some spatial coherence. We call this method a“Local window” method in this experiment.
The two methods based on local window and the global
PCA method have individual drawbacks. The global PCA
method does not have flexibility to represent variations inspectra, and the local window method does not consider
any color similarity in each window. To include both local and global characteristics, another idea is to cluster thehigh-resolution pixels based on their RGB values, and thenestimate a separate basis for each color cluster individually.
This “RGB clustering” method assumes that for the same
scene with the same RGB color value, it should produce
the same hyperspectral values. Although this assumption
is limited in some situations, we find that it leads to better
estimates than simple PCA methods.
Refinement by back-projection If the reconstructed image Ẑ does not satisfy the observation equation Prgb [Ẑ] =Yrgb , it can be further refined using via backprojection,
an interative method for finding a minimum-norm perturbation of Ẑ such that the observation equation is satisfied. This approach is often used in image super-resolution
methods [29]. In image super-resolution, the reconstructedhigh-resolution image, after down-sampling, needs to be thesame as the original low resolution input. In our case, we
can also incorporate the second constraint Phs [Z] = Yhs tofurther improve the result. In our experiments, we find thatfor the simple PCA methods introduced above, backprojection bring Z significantly closer to the ground truth.

5.2. Performance and Accuracy
We have conducted experiments using a spectral image
database described in [33], which provides spectral imagesof 32 scenes, sampled every 10 nm from 400 nm to 700 nm,
by using a liquid crystal tunable filter, VariSpec [31] witha black/white cooled CCD camera, Apogee Alta U260 [3].
Images were taken with fixed focus, in 16-bit PNG format,
and the resolution is 512 by 512 pixels.
To simulate a low-resolution camera, we down-sampled
them to 16 by 16 pixels. The RGB camera response were
simulated by integrating them over the wavelength, using
the filter characteristics of DSLR camera, Nikon D1. We

Input images: Balloons and Beads examples.

Ground truths.

Reconstruction using component substitution method [2].

Reconstruction using matrix factorization and refinement.

Error images of the matrix factorization and refinement.
Figure 2. Spectral images. The top row: Input images, sampled at 460, 550 nm, and the RGB image, from left to right. The first three
images are from Balloons example. The remaining images are the inputs of Beads. The second row: Ground truths, sampled at 460, 550,
and 620 nm. The order is the same in the following rows. The third row: Reconstructed results of component substitution method [2]. The
fourth row: Results of the proposed method. The bottom row: Error images of the proposed method.

used 8 examples in the database, and their simulated RGB
images are shown in Fig. 1. Down-sampled images areshown in the top row in Fig. 2.
To evaluate the performance of the proposed method, we
implemented the following five methods: (1) Component
substitution method [2], (2) Global PCA, (3) Local window, (4) RGB clustering methods (described in Sec. 5.1),
and the proposed method - (5) Matrix factorization method(described in Sec. 4.) Then, we apply a refinement method
to the initial estimates from these methods by the back-projection described in Sec. 5.1 For the RGB clustering,
we used k-means clustering with k = 10, and used chromaticity and spatial coordinate for the feature space. For
the matrix factorization method, we used 0.5 for ε in Eq. (6),and used the top 6 bases in A that contribute to each lowresolution spectrum, when doing the reconstruction.
The RMSE (Root Mean Squared Error) between the reconstructed spectral images and the ground truths (original data) are summarized in Table 1. Each row shows the
RMSE of the initial estimate of each method, and the RMSESpectral value120

g truthgloballocalwin
rgbclst
fact

100
80
60
40
20
0
400

500

600

700

Wavelength
80

200
150

g truthgloballocalwin
rgbclst
fact

100
50

Spectral value

Spectral value

250

0g truthgloballocalwin
rgbclst
fact

60
40
20
0

400

500

600

700

400

Wavelength

500

600

700

Wavelength

SpongesSpectral value

150

g truthgloballocalwin
rgbclst
fact

100

50

and 620 nm. The third row shows the reconstructed resultsof the component substitution method [2], which suffersfrom the large resolution difference between the hyperspectral and RGB images. The fourth and the bottom rows show
the results and the errors of the matrix factorization method.
The positive errors are shown in red, and negative ones arein blue. The proposed method succeeds to recover the details while maintaining the spectral information. The errors
are large around specular pixels that changes rapidly withrespect to the image resolution.
Fig. 3 shows the reconstructed spectra of three pixels in
Sponges and Fake and real peppers. The ground truth isshown as a bold line, and the reconstructed spectra of (1)Global PCA, (2) Local window, (3) RGB clustering and (4)Matrix factorization are also shown. Global PCA tends todistort the spectra due to the gross assumption that every
spectra can be represented by three dimensional basis functions. All the methods perform reasonably well, but above
all, the matrix factorization method provides the best results.

0
400

500

600

700

5.3. Real-world examples

Wavelength
140

100

Spectral value

100

80
60

Spectral value

g truthgloballocalwin
rgbclst
fact

120

40
20
0

g truthgloballocalwin
rgbclst
fact

80
60
40
20
0

400

500

600

Wavelength

700

400

500

600

700

Wavelength

Peppers
Figure 3. Comparison of spectra.

Lens

Variable filter
Board camera

Adjustable slit

Translation stage

Figure 4. Hyperspectral camera.

after the back-projection refinement. The RMSE values areshown in the range of 8-bit images. The average of the RMSEs are 16.1, 7.3, 7.8, 6.6 and 5.3 (6.3, 2.9, 3.1, 2.6, and
2.1 %) of 255 for CSM, Global PCA, Local window, RGB
clustering and Matrix factorization methods. The numbers
show the effectiveness of the proposed method. The tablealso shows that back-projection further improves the reconstruction error of the initial estimate.
The examples of the reconstructed images are shown in
Fig. 2. The first and second rows show the inputs and theground truths of Balloons and Beads examples, of 460, 550,

Experimental setup In order to obtain real image data tovalidate our hybrid frame-work we need two types of images. The first is high-spatial resolution RGB camera withknown spectral and intensity response functions, and thesecond is low spatial resolution, hyperspectral camera.
To obtain high-resolution RGB images, we used the4008 × 2672 Lumenera LW11509 camera with a long focal
length lens. This camera uses the Kodak KAI-11002 sensor that has large pixel, linear response function and known
spectral response for the RGB channels. The camera wasused with an integral Schott BG40 as an IR cutoff filter.
To obtain hyperspectral images, we built the device
shown in Fig. 4. This device uses variable linear band passfilter (Schott Veril 60), in conjunction with an adjustable slit
(Thorslab VA100/M) for bandwidth control and a CMOS
B/W sensor with a known response function (Point Grey
Firefly MV set to 640 × 480 resolution). The position of
the filter is controlled by a computerized translation stage(Zaber zaber KT-LSM100A). This setup is more economical than a tunable filter that are significantly more expensive, and provides better control than uncontrolled modulation imaging [27] because the filter is stationary during
image capture at any desired exposure time. We calibrated
our device using several narrow band filters. Because the repeatability of the translation stage is very accurate, we only
need to calibrate once. The camera can sample spectra at
every 5 nm from 415 nm to 730 nm. We also used several
narrow band filters to capture short wavelengths under 450
nm to ensure reliable measurement.
To capture images from the same view point, a beam
splitter is usually used. However, since we did not need tocapture them simultaneously, we simply replaced the cam-

Input RGB

Input (550nm)

Estimated (550 nm)

Input (620 nm)

Estimated (620 nm)

Figure 5. The figures show two zoomed views of the high-resolution image: The input and estimated spectral images of 550 and 620 nm.

era while maintaining the same view point. The slight difference of viewpoints were compensated by using the affine
transformation.
Results The input and reconstructed images of a printed
painting is shown in Fig. 5. The figure shows two zoomed
views of the painting. The high resolution RGB camera
can visualize the printing pattern, which is neither visible
to the human eyes nor to the low-resolution camera. The
proposed method recovers spectral images of the painting
in details, and provides visually pleasing results as shownin the figure. With the proposed method, highly detailed
spectral information is acquired in a very efficient manner,both in terms of the acquisition time and the cost of theequipment.

6. Discussion
As shown in the experiments, the matrix factorization
method and the RGB clustering method perform better thanthe others. This may not be surprising, since the two methods are similar in a sense that they utilize a kind of segmentation in spectra/RGB domains. They may be further
improved by jointly using those sources of information.
The distribution of the reconstruction errors of the proposed method is unique compared to those of the PCAbased methods. The proposed method sometimes fails toreconstruct specific colors, while errors are equally distributed to many colors when PCA-based methods wereused. PCA-based methods seem to suffer more than the
proposed method, because they discard the spectral information that lies in more than three dimensions.
The proposed method has more advantages in contrast with interpolation/super-resolution or pan-sharpening
methods, since these methods can usually only magnify animage up to a factor of 8, but our proposed method can magnify the hyperspectral image 32 times larger (32 × 32 resolution). Images with such a large magnification factor arealmost impossible to be reconstructed correctly by the current state-of-the-art methods.

There are limitations with the proposed method. The
method sometimes produces errors in some bands, whenmore than a few number of basis functions are required toreconstruct the full spectral image. This is a general limitation for all methods that assume a small fixed number ofbasis functions. However, in scenes that are composed ofa limited number of materials, our results clearly surpassthe results of all other straightforward methods. Thus, ourmethod has an advantage over these methods.
The refinement by the back-projection was not significant for the results of the matrix factorization method.
This is because the downsampled values from the recovered spectra are already very close to those in the original
RGB/spectral images when using the method.
The issue with the matrix factorization method is thecomputational time. For 4008 × 2672 images, it takes several hours to do the factorization, and several hours to do thereconstruction, using a high-end machine with eight cores.
The current implementation uses Matlab 7.0. We plan to
speed up the process by using GPU or multi-thread programming. There are parameters that have to be chosenwith the method. The epsilon depends on the noise level,
and the number of bases that contribute to a pixel is chosenempirically in the paper, but is fixed for all examples. We
consider that the latter can be roughly determined as long as
the convergence of the factorization is ensured.

7. Conclusion
We have proposed an efficient method to acquire highresolution hyperspectral images, by jointly using a highresolution RGB camera with a low-resolution hyperspectral camera. Assuming that only a few materials contributeto each pixel in the hyperspectral image, our method finds
the optimal set of basis functions by factorizing the hyperspectral image into the basis and its sparse coefficients via`1 -norm minimization. The spectrum at high-resolution isthen reconstructed using the estimated basis and the RGB
values. Comparison between the method and several simple approaches shows the effectiveness of the method. The

method enables to capture high-resolution hyperspectral
image at reasonable cost and acquisition time.

Acknowledgement
This work was in part supported by Microsoft Research
CORE 6 Project 2010.

References[1] M. Aharon, M. Elad, and A. Bruckstein. The K-SVD:
An algorithm for designing overcomplete dictionaries forsparse representation. IEEE Trans. on Signal Processing,51(11):4311–4322. 3[2] B. Aiazzi, S. Baronti, and M. Selva. Improving component
substitution pansharpening through multivariate regressionof ms+pan data. IEEE Trans. Geoscience and Remote Sensing, 45(10):3230–3239, 2007. 2, 3, 4, 5, 6
[3] Apogee. http://www.ccd.com/alta u260.html. 4[4] A. Bruckstein, D. Donoho, and M. Elad. From sparse solutions of systems of equations to sparse modeling of signals
and images. SIAM Review, 51(1):34–81, 2009. 2[5] A. Castrodad, Z. Xing, J. Greer, E. Bosch, L. Carin, and
G. Sapiro. Discriminative sparse representations in hyperspectral imagery. IMA Preprint #2302, University of Minnesota, 2010. 2[6] P. S. Chavez, S. C. Sides, and J. A. Anderson. Comparisonof three different methods to merge multiresolution and multispectral data: Landsat tm and spot panchromatic. Photo.
Eng. Rem. Sens., 57(3):295–303, 1991. 2[7] J. Cohen. Dependency of the spectral re ectance curves ofmunsell color chips. Psychon. Sci., 1:369–370, 1964. 4[8] H. Du, X. Tong, X. Cao, and S. Lin. A prism-based system
for multi-spectral video acquisition. In ICCV, 2009. 2[9] S. C. Gebhart. Liquid-crystal tunable filter spectral imaging
for discrimination between normal and neoplastic tissues inthe brain. PhD thesis, Vanderbilt University, 2006. 2[10] Q. Geng, H. Wang, and J. Wright. On the local correctness of`1 -minimization for dictionary learning. preprint, available
at http://arxiv.org/abs/1101.5672, 2011. 3[11] R. Gribonval and K. Schnass. Dictionary identification:
Sparse matrix factorization via `1 -minimization. IEEE
Transactions on Information Theory, 56(7), 2010. 3[12] F. H. Imai and R. S. Berns. High-resolution multi-spectral
image archives: a hybrid approach. In Color Imaging Conference, pages 224–227, 1998. 2[13] N. Keshava and J. Mustard. Spectral unmixing. Signal Processing Magazine, 19(1):44–57, 2002. 1[14] J. Kopf, M. F. Cohen, D. Lischinski, and M. Uyttendaele.
Joint bilateral upsampling. ACM Trans. Graphics, 26(3):96–
1–96–5, 2007. 2[15] J. Mairal, F. Bach, J. Ponce, G. Sapiro, and A. Zisserman.
Non-local sparse models for image restoration. In ICCV,2009. 2[16] L. T. Maloney and B. A. Wandell. Color constancy: a method
for recovering surface spectral reflectance. JOSA, 3(1):29–
33, 1986. 4[17] Y. Moudden, J. bobin, J. Starck, and J. Fadili. Blind source
separation with spatio-spectral sparsity constraints – application to hyperspectral data analysis. In Proc. Int’l Conf. on
Ind. Comp. Anal. Sig. Sep., 2009. 2[18] H. V. Nguyen, A. Banerjee, and R. Chellappa. Tracking via
object reflectance using a hyperspectral video camera. In
Proc. CVPR Workshop, 2010. 1[19] B. Olshausen and D. Field. Sparse coding with an overcomplete basis set: A strategy employed by v1? Vision Research,
37:3311–3325, 1997. 3[20] J.-I. Park, M.-H. Lee, M. D. Grossberg, and S. K. Nayar.
Multispectral imaging using multiplexed illumination. In
ICCV, 2007. 2[21] J. P. S. Parkkinen, J. Hallikainen, and T. Jaaskelainen. Characteristic spectra of munsell colors. JOSA, 6(2):318–322,
1989. 4[22] K. Pearson. On lines and planes of closest fit to systems ofpoints in space. Phil. Mag., 2:559–572, 1901. 3[23] Y. Pfeffer and M. Zibulevsky. A micro-mirror array based
system for compressive sensing of hyperspectral data. Technion Technical Report 2010-10, 2010. 2[24] A. Ribes, F. Shmitt, R. Pillay, and C. Lahanier. Calibrationand spectral reconstruction for crisatel: an art painting multispectral acquisition system. J. Im. Sci. Tech., 49:563–573,
2005. 2[25] J. Romberg. Imaging via compressive sampling. IEEE Signal Processing Magazine, 25(2):21–31, 2008. 2[26] R. Rubenstein, A. Bruckstein, and M. Elad. Dictionaries forsparse representation modeling. Proceedings of the IEEE,
98(6):972–982, 2010. 3[27] Y. Schechner and S. Nayar. Uncontrolled Modulation Imaging. In CVPR. 6[28] Y. Y. Schechner and S. K. Nayar. Generalized mosaicing. In
ICCV, 2001. 2[29] J. Sun, N. N. Zheng, H. Tao, and H. Y. Shum. Image hallucination with primal sketch priors. In CVPR, 2003. 4[30] D. Y. Tzeng and R. S. Berns. A review of principal component analysis and its applications to color technology. ColorRes. Appl., 30(2):84–98, 2005. 4[31] Varispec. http://www.cri-inc.com/products/varispec.asp. 4[32] M. Yamaguchi, H. Haneishi, and N. Ohyama. Beyond redgreen-blue (rgb): Spectrum-based color imaging technology.
J. Im. Sci. Tech., 52(1):010201–010201–15, 2008. 2[33] F. Yasuma, T. Mitsunaga, D. Iso, and S. K. Nayar. Generalized assorted pixel camera: Postcapture control of resolution,
dynamic range, and spectrum. IEEE Trans. IP, 19(9):2241–
2253, 2010. 4[34] K. J. Zuzak, R. P. Francis, E. F. Wehner, J. Smith, M. Litorja,
D. W. Allen, C. Tracy, J. Cadeddu, and E. Livingston. Dlp
hyperspectral imaging for surgical and clinical utility. In
Proc. SPIE, 2009. 2[35] A. Zymnis, S. Kim, J. Skaf, M. Parente, and S. Boyd. Hyperspectral image unmixing via alternating projected subgradients. In Proc. of Asilomar Conference on Signals, Systems
and Computers, pages 1164–1168, 2007. 2