Fourth-person Captioning: Describing Daily Eventsby Uni-supervised and Tri-regularized Training
Kazuto Nakashima

Yumi Iwashita

Akihiro Kawamura, Ryo Kurazume

Graduate School of ISEE
Kyushu University
Fukuoka, Japank nakashima@irvs.ait.kyushu-u.ac.jp

Jet Propulsion Laboratory
California Institute of Technology
Pasadena, CA, USA
Yumi.Iwashita@jpl.nasa.gov

Department of ISEE
Kyushu University
Fukuoka, Japan{kawamura, kurazume}@ait.kyushu-u.ac.jp

Abstract—We aim to develop a supporting system which
enhances the ability of human’s short-term visual memory in an
intelligent space where the human and a service robot coexist.
Particularly, this paper focuses on how we can interpret andrecord diverse and complex life events on behalf of humans, froma multi-perspective viewpoint. We propose a novel method named“fourth-person captioning”, which generates natural language
descriptions by summarizing visual contexts complementarily
from three types of cameras corresponding the first-, second, and third-person viewpoint. We first extend the latest image
captioning technique and design a new model to generate asequence of words given the multiple images. Then we provide aneffective training strategy that needs only annotations supervising
images from a single viewpoint in a general caption dataset andunsupervised triplet instances in the intelligent space. As the three
types of cameras, we select a wearable camera on the human, arobot-mounted camera, and an embedded camera, which canbe defined as the first-, second-, and third-person viewpoint,respectively. We hope our work will accelerate a cross-modal
interaction bridging the human’s egocentric cognition and multiperspective intelligence.
Index Terms—Intelligent space, image captioning

I. I NTRODUCTION
Intelligent space, the room or the area that is equipped withvarious sensors or cameras, has been widely studied in the
robotics community because of its feasibility of human-robot
coexistence [1]–[3]. Although it is difficult for a stand-alone
robot to observe the dynamic environment and operate diverse
service tasks for the humans with only on-board sensors,
intelligent space enables it to expand its observation area.
Moreover, the studies on intelligent space aim not only tocompensate their sensing ability but also to process the daily
streamed data and explore the information structure on behalfof the robot [2], [3].
Here we address the problem in which how the intelligent
space can be used to augment the human’s short-term visualmemory. Some studies [2], [3] focused on the human-centeredsystem as an external memory in the intelligent space. Niitsuma et al. [2] proposed a user interface which enables humans
to virtually arrange the computerized information to the real
3D space and to retrieve them by pointing. In our previous
work [3], we developed a glasses-type interface where theusers can know about the household items managed by theintelligent space, by passing object keywords or just looking

First-person vision

Second-person vision

Third-person vision

Fig. 1: Examples of the first-person images (left column),
the second-person images (middle column), and third-person
images (right column) in the intelligent space.

the shelves. Another approach to achieve the augmented
visual memories is to pool the autobiographical videos in adaily life, extract the latent semantics, and configure a video
retrieval system with a high-affinity interface for humans [4].
Particularly, the first-person videos captured from the user’s
egocentric viewpoint have been attracting increasing interests
for that purpose. Miyanishi et al. [5] proposed a gesture-based
video retrieval system. Nakayama et al. [6] proposed a goggletype system which retrieves the past images with object label
queries. Meanwhile, Fan et al. [7] adopted the encoder-decoder
neural network [8] to generate natural language description for
the first-person image and developed a retrieval system. The
natural language sentence given to the image explains not only
the related objects but also the relationship between objects.
Compared to the gestures and the object labels, it is literally anatural way for humans to generate flexible queries, e.g., theycan easily contain temporal terms. Due to the nature that thefirst-person videos contain undergoing hand manipulations andsaliency objects, the first-person videos have been used for not
only the automated lifelogging [5]–[7] but also wearer’s action
recognition [9], [10] and so on. The first-person viewpoint
can now easily be acquired by wearable devices, however, theview field is narrow and visual information about the wearer’ssurrounding fragmentarily appears.
Thus we focus on the multi-perspective vision we can leverage in the intelligent space. In the human-robot coexistence
situation, not just the user’s egocentric viewpoint, a robotmounted camera to monitor the human closely can also be
used. We define it as the second-person viewpoint. Moreover,the intelligent space generally has embedded cameras to observe the comprehensive state. We define this type of camera
as the third-person viewpoint. In our previous work [11], we
proposed “the fourth-person viewpoint” which complementarily integrates three type of viewpoints. We assumed thateach of these has unique visual concepts and can be merged
to compensate incomplete information. It aimed to interpret
daily events precisely like a book reader who picks up multiperspective sentences and appreciates the story.
In this paper, we present a novel method to describe daily
events with a natural language by using the fourth-person
viewpoint in the intelligent space. The triplet images via the
fourth-person viewpoint have unique resolutions and uniquevisible objects, which may raise the likelihood of the precise
caption generation. To our knowledge, we are the first to generate image captions from the multi-perspective viewpoints;
the first-, second-, and third-person viewpoints.
II. R ELATED W ORK
In this section, we describe relevant research background.
Most researches on the fusion of the multi-perspective
images focused on labeling of co-occurrence events related
to multiple viewpoints. Yonetani et al. [10] improved therecognition of interactive actions/reaction from a pair of thefirst- and second-person viewpoints. Fan et al. [12] proposeda method to identify the first-person video from some camera
wearers in the third-person videos. Other works leveraged themultiple videos for cross-view action recognition [13] andevent detection [14]. These studies focused on the classification/identification of human actions and have not reached acomprehensive description including handling objects and thecontext.
On the other hand, in recent years, the task of generating image captions expressing visual information withnatural language sentences has developed rapidly, motivatedby advances in deep neural networks [7], [8], [15]. Image
captioning is the primary form of understanding real-world
scenes and is indispensable for a high-affinity interface inthe future intelligent systems. However, conventional image
caption research did not explicitly consider the modality byperson, and handled images of all viewpoints in a single input
model. Some studies [7], [15] focused on the first-person
images/videos for image captioning task. The work by Fan
et al. [7] is most closely related to ours. They generated acaption from the first-person image sequences, in hopes ofautomatic lifelogging. They gathered the first-person images
from a wearable camera, construct an annotated dataset, andachieved to generate the “first-person” sentences. However, it
would take too much cost to newly annotate images. Even thework [7] above used a common dataset which is composed of

𝑆

𝑆

LSTM

LSTM

FCs

CNN

𝐼

Concat
CNN

CNN

CNN

𝐼"

𝐼#

𝐼$

Fig. 2: A basic architecture for generating a sequence ofwords S given a single image I (left) [8] and naı̈ve extension
for multiple inputs I1 , I2 , I3 (right). The “CNN”, “LSTM”,“FCs”, and “Concat” denote a convolutional neural network,
a long shot-term memory module, fully-connected layers, andconcatenating operation, respectively.

random images. It can be still more a crucial problem for ourmuti-perspective situation. Therefore, we tackle the problem
by taking a strategy to solve our objective with only singleperspective supervision.
III. C APTIONING M ODELIn this section, we describe the models that we use for thecaption generation from three types of images. Our model isextended version of the encoder-decoder model [8] as shownin Fig. 2. We add a fully-connected fusion layer which projects
the triplet image features produced by the encoder to a single
representation vector.
A. Image Encoder
Our model takes three types of raw images from a wearable
camera, a robot, and an embedded camera. We use a convolutional neural network (CNN) pre-trained on the large-scale
image dataset such as ImageNet [16] in order to encode themas a set of viewpoint-wise feature vectors fi , each of which isa D1 -dimensional global image representation.
B. Fusion Layer
The triplet image features fi ∈ RD1 are simply concatenatedand fed into a fully-connected layer to project to a multimodal
space RD2 which is followed by the caption decoder.
f′ = W · f
Tf = [f1 , f2 , f3 ]

(1)
(2)

where W ∈ RD2 ×3D1 is learned parameters. The f ′ isfollowed by a non-linearity and Batch Normalization [17]. We
employ a hyperbolic tangent for the non-linearity.

Reconstruction loss ℒ123

Image-caption loss ℒ43+

𝐿 $distance−Σ log 𝑝- 𝑠LSTM
SharedFCs

FCs

ConcatSwitch View

CNN

CNN

CNN

𝐼"

𝐼$

𝐼%

Shared

CNN

𝐼

Triplet images: 𝐼", 𝐼$ , 𝐼%

Image-caption pair: 𝐼, 𝑆

Tri-regularization

Uni-supervision

Fig. 3: Schematic illustration of training of our multiperspective captioning model. In the uni-supervision stream,
the visual information (solid lines) is fed into the LSTM cellonce at the first time step.

C. Recurrent Caption Decoder
We use a recurrent network with a long short-term memory
cell (LSTM) to generate words of a caption S = {s0 , . . . , sT }.
At each time step t, the LSTM updates its hidden state ht ,given previous hidden state ht−1 and an input vector xt .
ht = LSTM(ht−1 , xt )(3)

The multi-perspective feature f ′ generated by the fusion
layer is fed into the LSTM as x0 at the first step t = 0. At
the following time steps t ≥ 1, a word st−1 is embedded to a
D2 -dimensional space as xt .
xt = We · st−1

(4)

where We ∈ RD2 ×Ds is learned parameters. Note that ast ∈ RDs is a word vector in one-hot representation andits sequence starts with a special start token s0 . The state
ht is then multiplied by additional parameters Ws and fed tosoftmax(·) which produces a probability distribution pt overthe pre-defined vocabulary words. The next word st is definedby argmax(pt ).
IV. U NI - SUPERVISED AND T RI - REGULARIZED T RAINING
In this section, we address the problem on training the multiperspective model in our situation and describe our approach.
Traditionally, given an image I and a target word sequence
S = {s0 , . . . , sT } as groud truth, the model θ is trained byminimizing the sum of the crossentropy loss of the given
words, which equals to maximize the likelihood of the target
sequence.
T∑
LCE = −
log pt (st )(5)t=1

To compute the loss, we need to initialize the LSTM with
the instance of the fused image feature at the first step, as
described in the previous section. Therefore, for such an
end-to-end manner, we have to newly prepare a large-scale
dataset consists of pairs of the triplet image inputs and thecorrespoding caption outputs. However, it can be consideredthat the daily living environments are diverse regarding household items or room layouts and it would take too much costto construct a new dataset which enables to learn visualsemantic relationships and generalize every case. In this paper,
instead, we propose a novel training strategy to train themulti-perspective captioning model with a single perspective
supervision generally used to train captioning models.
To supervise the multi-perspective model with a general pairof a single image and caption candidates, we have several
options. The first approach is to duplicate the single image
to three inputs for the fusion layer and train the entire modelin the usual manner. This approach can sufficiently train thelanguage model but it may overfit to the “duplicated input”
situation which is totally different from the multi-perspective
vision. The second approach is to pool the image features, forexample by averaging or taking max values over the three
viewpoints, and feed into the LSTM. The feature poolingtechniques have been applied for unimodal images based on
the consistent contents, such as temporal pooling of framewise features in video recognition [18] and multi-view poolingin 3D shape recognition [19] whereas we focus on the multiperspective images. It can be easily applied various existing
methods and pre-trained models as is, however, some unique
attributes in each image may be diminished by pooling.
Consequently, we propose uni-supervised and triregularized training which aims to train the multi-perspectivemodel without fully-paired supervision (See Fig. 3). It
consists of the uni-supervision stream that leverages generalimage-caption pairs and the tri-regularization stream that
self-supervises by the triplet images. On the uni-supervision
stream, we first make a masked feature instead of the fin Eq. 2 by randomly select a view as an image featureand setting other views to zero (”Switch View” in Fig. 3).
Furthermore, all the nodes in the selected view are scaled byfactor 3. Note that we change this multiplexer mechanism
to an identity function during testing. Next, we propagate
through the remained networks and computes the crossentropy
loss LCE indicated in Eq. 5. Whereas, the tri-regularization
stream does not use the image-caption dataset but sets of
triplet images captured from the actual scenes. It configures
the autoencoder, containing the fusion layer as an encoding
part. This stream aims to preserve the information in threeviews and avoid simple averaging in the uni-supervision
stream. Thus we compute the reconstruction loss LM SE
defined by mean square error. From the CNNs to the fusion
layer, both streams share the parameters to be learned. The
objective is to minimize a weighted loss of LCE and LM SE .
L = LCE + λLM SE(6)

TABLE I: Caption scores on MSCOCO test splits.
Input

BLEU-4

METEOR

ROUGE-L

CIDEr-D

SPICE

SingleTriplet†

27.8
26.0

24.3
23.0

51.8
50.3

90.9
83.2

17.6
16.2

† Set an identical image to three inputs.

Fig. 4: Experiment room we use for data collection

V. E XPERIMENTS
A. Data
We use MSCOCO dataset [20] to train our model on theuni-supervision stream. We use the data splits from [21]which contains 113,287 images for training, 5,000 images forvalidation, and 5,000 images for testing. Each image has 5
captions. As a preprocessing of the sentences, we remove
punctuation and unify them in lower case. We prepare a<start> and a <end> tokens as special words and prune
the vocabulary by defining any words that have a count lessthan 5 as a special <unknown> word. The final vocabularycomprises 10,107 words.
Furthermore, to train our model on the tri-regularization
stream, we collected multi-perspective image sets in the experiment room shown in Fig. 4. The room is filled with various
items of furniture such as a desk, a couch, a plant, and a
television. We assumed that there are one resident and one
service robot in the room. To assemble and collect the triplet
images, a person wore a glasses-type wearable camera (Vuzix
M100 [22]) for streaming the first-person images to a cloudserver and acted some activities of daily living such as readinga book, walking around, and watching television. Moreover,
we collected the second- and third-person images, from a camera in the head of the service robot (Yaskawa SmartPalV [23])and cameras fixed on the wall, respectively. Each image is
360×480 resolution. We simultaneously recorded three type ofimages in every 5 seconds, with 8 different camera positions.
The final image sets contain 200 triplets for training and extra
triplets in several situations for testing. Example images areshown in Fig. 1.
B. Implementation Details
The parameters of image encoder are transferred fromResNet-152 [24] pre-trained on ImageNet [16] dataset up to
the global average pooling layer. Thus, given an arbitrarily
sized color image, the outputted image feature is a 2048dimensional vector. The fusion layer receives a concatenated6144-dim vector from the image encoder and projects it to2048-dim feature space (see Eq. 1). The input and hidden
state dimensions of the LSTM are set to be 512.
We compute the weighted loss from the crossentropy of
MSCOCO data and the self-supervised mean square error of
the triplet images, each of which is with a mini-batch of

16. We choose the weight of 1 for λ of Eq.6. We train ourmodel under the weighted loss objective using Adam optimizer
with a learning rate of 5 × 10−4 . We decay the learning rateby multiplying factor 0.8 in every 3 epochs. Moreover, we
employ ”Scheduled Sampling” [25] where the model feeds
back its own prediction to input at next step under the feedback
probability p. We initialize the factor p with 0 and increase by
0.05 every 5 epochs until it reaches 0.25. We select the final
model which achieves the best CIDEr-D score on duplicatedinput images made of MSCOCO validation set.
All algorithms are implemented in PyTorch [26] and run on
NVIDIA GeForce GTX Titan X.
C. Results
We compare our multi-perspective model with the basic
single-perspective model which receives only one image. We
herein denote the models as “Triplet” and “Single”, respectively. In Table I, we show some qualitative scores on the
MSCOCO test split with common metrics in natural language
processing tasks. We generated captions by greedy decoding.
Note that our scores here are not in the multi-perspective
scenario. It can be seen that the performance gap is smalleven with the random view switching and the regularization
during training our model.
In addition to caption generation, we visualize the internal
response in models by Grad-CAM [27]. Grad-CAM enables
us to see discriminative regions that strongly influence a
resulted caption, by weighting feature maps of a certain layer
with gradients derived from the score. We choose the lastconvolution layer of the image encoder for extracting featuremaps. Moreover, we extend the Grad-CAM calculation bycomputing word-wise log probability for each step instead ofsentence-wise log probability [27]. We normalize the gradientbased weights over three streams of all the steps.
We generated captions from our triplet images with (i)the single-perspective model which produces for each image(Single), (ii) the same model with a pooled feature (SinglePool), and (iii) our proposed multi-perspective model (Triplet).
We used beam search approach to decode words with a beamwidth of 5. The example of generated captions and the GradCAM visualization is shown in Fig. 5. In the situation of Fig. 5,
the man sitting on the couch is reading the book, otherwise,
the robot is facing to the man and they are in the living room.
(i) As shown in Fig. 5(a)(b)(c), given the first-, second-, andthird-person images, the single-perspective model produced “aperson holding a pair of scissors in their hand”, “a woman
sitting on a couch with a cat in her lap”, and “a man sitting ina chair in a living room”, respectively. Although each caption
deviates from the actual scenes, they captured fragmentary

visual concepts, for instance, the fact that the actor holding aobject, which has appeared in the first-person vision. (ii) As
shown in Fig. 5(d), the feature-pooling method produced “awoman sitting on the floor playing a video game”. It succeeded
to describe the sitting man but the remained sentence is totally
incorrect. According to the visualization, the phrase ”playing avideo game” was affected by the region around the robot. (iii)
Finally, as shown in Fig. 5(e), our proposed multi-perspectivemodel produced “a woman sitting on a couch using a laptop
computer”. The result was still inappropriate but succeeded
to describe the woman sitting on a couch from the triplet
images with only the single supervision, and to strongly andsimultaneously activate in the couch regions on the secondand third-person images. Additionally, we provide other resultswith four types of models in Fig. 6.
VI. C ONCLUSION AND D ISCUSSION
In this paper, we have proposed a novel image captioning
approach using a multi-perspective viewpoint, i.e., the firstperson image from a wearable camera, the second-person
image from a camera mounted on a robot, and the third-person
image from an embedded camera in the intelligent space. To
generate captions from the triplet images, we extended thebasic encoder-decoder network and introduced uni-supervised
and tri-regularized training which enables us to train themodel with only single-perspective annotations. Our model
succeeded to learn to generate caption from the triplet images
under the training strategy we proposed. Although we could
confirm some improvement in some samples, even the resultsfrom single images contained inappropriate words. It can beconsidered that the image quality or domain gap between
datasets caused the problems. Future work includes constructing an annotated test set for evaluation and demonstrate theeffectiveness quantitatively.
ACKNOWLEDGMENT
This work was partially supported by JSPS KAKENHI
Grant Number JP26249029 and JST CREST Grant Number
JPMJCR17A5, Japan.
R EFERENCES[1] J.-H. Lee and H. Hashimoto, “Intelligent space―concept and contents,”
Advanced Robotics, vol. 16, no. 3, pp. 265–280, 2002.
[2] M. Niitsuma and H. Hashimoto, “Spatial memory as an aid system forhuman activity in intelligent space,” IEEE Transactions on Industrial
Electronics, vol. 54, no. 2, pp. 1122–1131, 2007.
[3] R. Kurazume, Y. Pyo, K. Nakashima, A. Kawamura, and T. Tsuji,“Feasibility study of iort platform big sensor box ,” in Proceedings ofthe IEEE International Conference on Robotics and Automation (ICRA).
IEEE, 2017, pp. 3664–3671.
[4] M. Bolanos, M. Dimiccoli, and P. Radeva, “Toward storytelling fromvisual lifelogging: An overview,” IEEE Transactions on Human-Machine
Systems, vol. 47, no. 1, pp. 77–90, 2017.
[5] T. Miyanishi, J.-i. Hirayama, Q. Kong, T. Maekawa, H. Moriya, andT. Suyama, “Egocentric video search via physical interactions,” inProceedings of the Thirtieth AAAI Conference on Artificial Intelligence,ser. AAAI’16. AAAI Press, 2016, pp. 330–336. [Online]. Available:
http://dl.acm.org/citation.cfm?id=3015812.3015861[6] H. Nakayama, T. Harada, and Y. Kuniyoshi, “Ai goggles: Real-time
description and retrieval in the real world with online learning,” inCanadian Conference on Computer and Robot Vision (CRV). IEEE,
2009, pp. 184–191.

[7] C. Fan and D. Crandall, “Deepdiary: Automatically captioning lifelogging image streams,” in European Conference on Computer VisionInternational Workshop on Egocentric Perception, Interaction, and Computing (EPIC), 2016.
[8] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and tell: Lessons
learned from the 2015 mscoco image captioning challenge,” IEEE
Transactions on Pattern Analysis and Machine Intelligence, vol. 39,
no. 4, pp. 652–663, April 2017.
[9] M. S. Ryoo and L. Matthies, “First-person activity recognition: What are
they doing to me?” in Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition (CVPR). IEEE, 2013, pp. 2730–2737.
[10] R. Yonetani, K. M. Kitani, and Y. Sato, “Recognizing micro-actions andreactions from paired egocentric videos,” in Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition (CVPR), 2016,
pp. 2629–2638.
[11] K. Nakashima, Y. Iwashita, P. Yoonseok, A. Takamine, and R. Kurazume, “Fourth-person sensing for a service robot,” in Proceedings ofthe IEEE Conference on Sensors, 2015, pp. 1–4.
[12] C. Fan, J. Lee, M. Xu, K. Kumar Singh, Y. Jae Lee, D. J. Crandall, and
M. S. Ryoo, “Identifying first-person camera wearers in third-person
videos,” in Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition (CVPR), July 2017.
[13] J. Liu, M. Shah, B. Kuipers, and S. Savarese, “Cross-view action
recognition via view knowledge transfer,” in Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition (CVPR).
IEEE, 2011, pp. 3209–3216.
[14] V. Ramanathan, J. Huang, S. Abu-El-Haija, A. Gorban, K. Murphy, and
L. Fei-Fei, “Detecting events and key actors in multi-person videos,” inProceedings of the IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), 2016, pp. 3043–3053.
[15] M. Bolaños, Álvaro Peris, F. Casacuberta, S. Soler, and P. Radeva,“Egocentric video description based on temporally-linked sequences,”
Journal of Visual Communication and Image Representation, vol. 50,
pp. 205–216, 2018. [Online]. Available: http://www.sciencedirect.com/
science/article/pii/S1047320317302316[16] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in Proceedings of theIEEE Conference on Computer Vision and Pattern Recognition (CVPR),
Miami, FL, USA, June 2009, pp. 248–255.
[17] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep
network training by reducing internal covariate shift,” in Proceedingsof the International Conference on Machine Learning (ICML), 2015,
pp. 448–456.
[18] M. S. Ryoo, B. Rothrock, and L. Matthies, “Pooled motion features
for first-person videos,” in Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), 2015, pp. 896–904.
[19] H. Su, S. Maji, E. Kalogerakis, and E. Learned-Miller, “Multi-view
convolutional neural networks for 3d shape recognition,” in Proceedingsof the IEEE Conference on Computer Vision and Pattern Recognition(CVPR), 2015, pp. 945–953.
[20] X. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollár, andC. L. Zitnick, “Microsoft coco captions: Data collection and evaluation
server,” arXiv preprint arXiv:1504.00325, 2015.
[21] A. Karpathy and L. Fei-Fei, “Deep visual-semantic alignments forgenerating image descriptions,” in Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 3128–
3137.
[22] “Vusix M100,” https://www.vuzix.com/Products/m100-smart-glasses.
[23] “Yaskawa SmartPalV,” https://www.yaskawa.co.jp/en/.
[24] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image
recognition,” in Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition (CVPR), 2016, pp. 770–778.
[25] S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer, “Scheduled sampling
for sequence prediction with recurrent neural networks,” in Advances in
Neural Information Processing Systems (NIPS), 2015, pp. 1171–1179.
[26] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin,A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation inpytorch,” 2017.
[27] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and
D. Batra, “Grad-cam: Visual explanations from deep networks via
gradient-based localization,” in Proceedings of the IEEE International
Conference on Computer Vision (ICCV), Oct 2017.

a(0.57)

Grad-CAM

person(0.49)

holding(0.56)

a(0.58)

pair(0.22)

of(0.91)

scissors(0.68)

in(0.52)

their(0.18)

hand(0.50)<end>(0.89)(a) The first-person viewpoint by the model Single
Grad-CAMa(0.75)

woman(0.52)

sitting(0.44)

on(0.81)

a(0.80)

couch(0.83)

with(0.35)

a(0.53)

cat(0.45)

in(0.14)

her(0.64)

lap(0.98)<end>(0.90)

tv(0.44)<end>(0.55)(b) The second-person viewpoint by the model Single
Grad-CAMa(0.66)

man(0.30)

sitting(0.15)

in(0.48)

a(0.75)

chair(0.61)

in(0.26)

front(0.24)

of(0.92)

a(0.87)

Grad-CAMa(0.85)

woman(0.30)(c) The third-person viewpoint by the model Single
sitting
on
the
floor
playing(0.40)(0.60)(0.26)(0.91)(0.19)

a(0.42)

video(0.37)

game(0.89)<end>(0.73)

Grad-CAMa(0.83)

woman(0.42)(d) The fourth-person viewpoint by the model Single-Poolsitting
on
a
couch
using
(0.29)(0.55)(0.66)(0.59)(0.21)

a(0.78)

laptop(0.81)

computer(0.38)<end>(0.94)(e) The fourth-person viewpoint by the model Triplet (ours)

Fig. 5: Results from triplet images captured in the intelligent room. The first and the second figures from the left are input
images and sentence-wise Grad-CAM [27]. The following figures from the third are word-wise Grad-CAM, each of which iswith the predicted word and its probability. Regions highly contributing to predict the word appears in red color.
Example 1

Example 2Single-1: a close up of a person holding askate board, Single-2: a couple of people
that are sitting on a couch, Single-3: a groupof people sitting around a table in a room,
Single-Pool: a woman sitting in front of alaptop computer, Triplet: a man sitting ona couch in a living room

Example 3Single-1: a close up of a black and white
object, Single-2: a woman sitting at a deskwith a laptop, Single-3: a living room with
furniture and a flat screen tv, Single-Pool:a person sitting at a desk with a laptop,
Triplet: a desk with a computer and a lamp
on it
Example 4Single-1: a person holding a nintendo wii
game controller, Single-2: a woman sitting
at a table with a cup of coffee, Single-3: aliving room filled with furniture and a flat
screen tv, Single-Pool: a man sitting at atable in front of a laptop computer, Triplet: aman sitting at a desk in front of a computer

Single-1: a close up of a person holding acat, Single-2: a woman standing in front ofa refrigerator, Single-3: a living room filledwith furniture and a fire place, Single-Pool:a person sitting on a chair in front of a tv,
Triplet: a man standing in a kitchen next toa cat

Fig. 6: Generated examples, each of which is with input images (left) and outputted captions (right). The each image is a setof the first- (top), second- (middle), and third-person images (bottom). The ”Single-X” denotes a caption from the Xth-person
image. True positive terms are highlighted in blue. Some improvements by Triplet model (ours) can be seen in the examples
1 and 3. Whereas the examples 2 and 4 fail to generate appropriate captions.