# Enhanced Image Caption Generator Using Deep Learning

Sreeja Pinnapureddy, Abhishek Raparthy, Reethusree Suthari, A. Bhanuprasad, Shanmugasundaram Hariharan

► To cite this version:

Sreeja Pinnapureddy, Abhishek Raparthy, Reethusree Suthari, A. Bhanuprasad, Shanmugasundaram Hariharan. Enhanced Image Caption Generator Using Deep Learning. 6th International Conference on Computational Intelligence in Data Science (ICCIDS), Feb 2023, Chennai, India. pp.314-326, 10.1007/978-3-031-38296-3_24. hal-05466055

HAL Id: hal-05466055

https://inria.hal.science/hal-05466055v1

Submitted on 19 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
<figure><img src="image_3.png" /></figure>

# Enhanced Image Caption Generator Using Deep Learning

Sreeja Pinnapureddy(✉), Abhishek Raparthy(✉), Reethusree Suthari, A. BhanuPrasad, and Shanmugasundaram Hariharan

Department of Computer Science and Engineering, Vardhaman College of Engineering,
Hyderabad, India

a.bhanuprasad@vardhaman.org

**Abstract.** Image captioning has evolved into one of the most extensively used technologies in the modern period. Image captioning is the course of creating a description for an image. Recognizing the significant objects, their characteristics, and connections between objects in the image is required. It generates sentences that are both syntactically and semantically correct. In this project, we thoroughly examine an image caption creation technique based on deep neural networks. The approach may produce an English sentence explaining the content of an image when given an input of an image. CNN, RNN, and sentence generation are the three parts of the method that is examined. A numerous deep neural network-based models are described for creating image captions, concentrating on the various RNN approaches and examining their impact on phrase synthesis.

**Keywords:** CNN · RNN · LSTM · Xception · Neural Networks · VGG16 · ResNet

## 1 Introduction

A long-standing challenge in artificial intelligence is teaching a computer system to recognize items and describe them by Natural Language Processing (NLP). Until recently, academics studying computer vision regarded this as an impossibility. Models are frequently constructed that will generate subtitles for a picture thanks to the rapid breakthroughs in deep learning strategies, the availability of enormous datasets, and processing capacity [1]. To understand the context of a picture and explain it in a native tongue, such as English or another vocabulary, image caption generation uses IP and NLP ideas. The efficient development of more accurate predictions has involved researchers [2].

To create a solid model, deep neural networks and ML methods. The Flickr 8k dataset was used, which includes five captions for each of the approximately 8000 example photographs in the collection. CNNs are utilized to extract features from the picture, and RNNs are used to produce text in natural language with respective to the image [3].
Instead of just identifying the items in the image for the first phase, we developed a different method of extracting features that would reveal even the smallest differences between two identical photos. By Using two-dimensional arrays of input photos, a computer system maps the visuals to captions or descriptive phrases [4].

The process of creating captions for photographs automatically has received a lot of attention recently. However, while benchmark datasets need quick, precise, and competitive evaluation methods to promote quick advancement, new datasets frequently spark significant innovation. Because it excels at a type of learning that could be extremely beneficial for real-world applications, deep learning has garnered a lot of attention. Unlabeled or unstructured data might be a goldmine for those who are interested in practical applications [5].

## 2 Literature Survey

There have been numerous attempts to address this issue, including template based methods that used picture classification—adding marks to things from a predetermined various classes and embedding them in a model pattern sentence—to address the issue. Recurrent neural networks, however, have been the subject of more recent study. RNNs are already widely used for a variety of NLP applications, including machine translation where a list of words is created. By creating a description for a picture word by word, the image caption generator expands the functionality of the same application [6]. Since a decade ago, there has been substantial research into content-based image retrieval techniques, which leverage visual contents to search photos from massive image databases. It is becoming more and more clear that an object recognition system must be tailored to certain applications, such as engineering and CAD designs [17].

Object detection utilizing static article class packages in the picture and modelling using linguistic statistics design was the starting point for the key issue in the development of image description. Primarily using CNN: CNN is a DL algorithm that takes a 2D matrix as input image, give weights and biases to various viewpoints and items in the photograph, and smart to identify from one to the other. While this design was useful for identifying the devices in a picture, it was unable to say us how those objects related to one another [7]. This study provides, A generative model in light of a profound repetitive design that joins ongoing improvements in PC vision and machine interpretation and efficiently generates words with meaning. Furthermore using an RNN: These networks have loops that enable information to persist. An RNN variant that can recognize long-term dependencies is the LSTM [8]. The selection of the appropriate model for the caption generating network was one of the biggest challenge that is encountered [9].

Using a multimodal CNN to compare phrases and images while taking into account their various levels of interaction. Picture CNNs to encode visual data, linking CNNs to simultaneously symbolise visual and textual data, and multilayer perceptions to rate interoperability of visual and textual data are the three types of components included in Ma's system. To accommodate for joint depictions of pictures and words, buzzwords, and statements, the authors employ several corresponding CNN versions. An ensemble of multidimensional CNN is used to determine the last matching score among a picture and a phrase [14].
Tokenized inscriptions and picture vectors are both inputs to an RNN block in the first case, while just the inscriptions are input in the second case, and the output is merged with the picture. Although the tests indicate that there is not very much of a distinction between the two models’ accurateness, it is good to choose the merge architecture because to its straightforward design, which results in fewer hidden states and quicker training. Additionally, the RNN network’s memory is used more effectively because the images are not cycled through iteratively [10].

# 3 Background Algorithms

A hierarchy of levels in an artificial neural network made for deep learning is used to carry out the machine learning process. The model depends on deep networks, in which the progress of data begins at the introductory level, where the model gets the hang of something straightforward and then passes its output to the network’s second layer while combining its input with something slightly more complicated and passing it on to layer three of the network. As each level in the network builds on the information it acquired from the ascending level, this process continues (Fig. 1).

<figure><img src="image_4.png" /><figcaption>Fig. 1. General CNN-RNN model [1]</figcaption></figure>

## 3.1 Convolutional Neural Network (CNN)

Convolutional Neural Networks are personalised deep neural networks that are proficient of handling data with input shapes corresponds to a 2D matrix. A simple 2D matrix can be used to represent images. CNN is essential while using photos.
It accepts an image as input, gives various elements and objects in the image weights and biases, then separates them out according to relevance. Similar to how the human brain recognizes objects in existence, the CNN employs filters to aid in feature learning and detect abstract notions such as blurring, edge identification, honing, and so forth (Fig. 2).

<figure><img src="image_5.png" /><figcaption>Fig. 2. CNN Architecture for Classifying Objects [2]</figcaption></figure>

An artificial neural network called a CNN has been successfully used to examine visual images. The neurons communicate similarly to the animal visual cortex and are motivated by biological systems. Support Vector Machine (SVM) approaches are utilised for categorization of such photos in [16] which discusses automatic recognition of cow images using CNN that enables to extract the essential components from the cattle images.

CNN is used in this research to translate an RGB image to a visual feature vector. Convolution, pooling, and fully-connected layers are the three most often utilized CNN layers. Additionally, the non-linear active function ReLU $f(x) = max(0, x)$ is utilized. ReLU is quicker than the conventional formula $f(x) = tanh(x)$. Overfitting is avoided by the usage of dropout layer. Each hidden neuron's output is set to zero with a probability of 0.5 by the dropout. The “dropped out” neurons are not involved in either the forward pass or the back propagation [2] (Table 1).

## 3.2 Recurrent Neural Network (RNN)

The mortal mind has developed in a manner that it can understand words that have come before and then generate new words while keeping those words in mind, creating perfect sentences. Basic neural networks are unable to accomplish this. Recurrent neural network developments, however, take care of this problem. They are networks with loops that, by utilizing their internal states to create a feedback loop, enable information to endure for a time [11].
<table><caption>**Table 1.** Architectures of CNN</caption><thead><tr><th>Year</th><th>CNN</th><th>Features</th><th>Importance</th><th>No. of layers</th><th>No. of parameters</th></tr></thead><tbody><tr><td>[4]</td><td>Google Net</td><td><ul><li>1 × 1 convolution</li><li>Inception module</li><li>Auxiliary Classifier for training</li></ul></td><td><ul><li>Image classification</li><li>Object recognition</li><li>Quantization</li></ul></td><td>22</td><td>4 million</td></tr><tr><td>[5]</td><td>VGG Net</td><td><ul><li>Has two networks i.e., VGG-16, VGG-19</li></ul></td><td><ul><li>Largescale Image Recognition</li></ul></td><td>16, 19</td><td>138 million</td></tr><tr><td>[6]</td><td>Res Net</td><td><ul><li>Skip Connection technique is used</li><li>Residual mapping</li></ul></td><td><ul><li>Efficient backbone model</li></ul></td><td>34</td><td>25 million</td></tr><tr><td>[1]</td><td>Xception</td><td><ul><li>Depth wise separable Convolutions</li><li>Takes the tenets of Inception for logical conclusion</li></ul></td><td><ul><li>Image recognition</li></ul></td><td>71</td><td>22 billion</td></tr></tbody></table>

Long Short Term Memory Networks, most basically referred as “LSTMs,” are a distinctive class of RNN that can acknowledge long-term dependencies. Their natural tendency is to retain knowledge for extended periods of time, and “gates” are used to control this behaviour. LSTMs can handle complete sequences, but RNNs can only process a single data point. Additionally, they can discover which data piece is crucial and which can be ignored. As a result, only pertinent data is sent on to the following layer (Fig. 3).

<figure><img src="image_6.png" /><figcaption>**Fig. 3.** Recurrent Neural Network Loop [12]</figcaption></figure>
The input gate, output gate, and forget gate are the three main gates in play. These gates choose whether to add a worth to the cell, yield the worth of the cell, or else forget the current cell price. Since the prior concealed states are sent to the following stage of the succession, the covered up states play a crucial function. The information that the brain network has previously seen is stored in the hidden state, which serves as its memory. As a result, it enables the neural network to operate similarly to a human mind attempting to construct phrases [8] (Fig. 4).

<figure><img src="image_7.png" /><figcaption>Fig. 4. There are four interconnected layers in the recurrent module of the LSTM [13]</figcaption></figure>

# 4 Proposed Work

The goal is to create a system that can accept a picture input as a layered cluster and produce an output that is a syntactically and grammatically sound sentence that describes the image.

## 4.1 Architecture

To accept a picture as info and result a subtitle, CNN + LSTM model is used. The final meaningful sentence is generated as a prediction by a "decoder" RNN using the introductory hidden state of a "encoder" RNN, which first traces the source phrase (that is of different lengths) and converts it to the constant-length matrix portrayal.

The photos and their related annotations are cleaned and prepped independently as part of the data pre treatment process. The Xception version of the Keras API, which is built on top of TensorFlow, is used for image pre processing. ImageNet is used to pre-train Xception. The tokenizer module in Keras is used to clean up the descriptions; this vectorizes the text corpus and stores the cleaned data in a different dictionary. Then,
a distinct index value is assigned to each word in the lexicon. You can use the Keras or
perform the aforementioned two pre processing steps yourself.

However, by pre-training a deep for a picture order errand and involving the last
concealed layer as contribution to the RNN parser for a photo classification mission and
utilizing the final hidden layer as the input to the RNN decoder which creates statements,
we can substitute this RNN with a strong CNN instead, which can create a grand portrayal
of the information picture by installing it as for a fixed-duration vector (Fig. 5).

<figure><img src="image_8.png" /><figcaption>Fig. 5. CNN-LSTM Architecture</figcaption></figure>

## 4.2 Dataset

The corpus that we used was the Flickr8K dataset [15]. There are 5 captions for each of the 8000 photos in the dataset. Understanding all the different circumstances is made easier by the single image's 5 captions. The dataset includes three prepared datasets: Flickr 8k trainImages.txt (6,000 photos). The pictures are chosen from 6 different Flickr clusters and don't feature possible famous people or locations. A lexicon of all distinctive words from each description is developed and will later be utilized to provide captions for test photographs. There's still more to our language is tokenized and given a special index value as part of the data pre processing procedure.

This is due to the fact that since a computer cannot understand ordinary English
words, numbers must be used to represent them. After that, tokens are kept in a pickle
file as a character flow, yet with total data needed to turn it back as the first type of
item. Next, we will determine the greatest size of the portrayal, which can be utilized in
upcoming stages, and the amount of terms in our vocabulary.

A feature vector, often known as a “feature,” is a mathematical value in matrix
structure that contains details about an object’s key attributes, such as the intensity value
of each pixel in our example image. We’ll eventually save these vectors in a pickle
file. To extract features from a pretrained model. It was trained using the well-known
Image net dataset, which contains millions of photos and more than a thousand distinct
classifications.

The photos and their related captions are cleaned and prepped independently as part
of the data pre treatment process. Xception implementation of the Keras API, which runs
ahead of TensorFlow, is used for image pre processing. ImageNet is used to pre-train
Xception. This allowed us to use transfer learning to train the images more quickly.
The tokenizer class in Keras is used to clean up the descriptions; this vectorizes the text
corpus and stores the cleaned data in a different dictionary. The words is then arranged
with a different index worth for each term.
## 4.3 Evaluation

To work comfortably and quickly, Google Colaboratory is used, a platform that offers free GPU/TPU handling power in contrast to our ordinary PCs, which can require hours to do tasks that a GPU can complete in a matter of minutes. First of all, load the dataset in the google drive and mount the drive into current working file. Install all the necessary packages for the project like, tensorflow, keras, xception etc.

A feature vector, often known as a "feature," is a mathematical worth in grid structure that contains details about an object's key attributes, such as the intensity value of each pixel in our example image. We'll eventually save these vectors in a pickle file. To extract features from a pretrained model, we'll be employing transfer learning in our model. The Xception model is a 71-layer CNN. It was trained using the well-known Imagenet dataset, which contains millions of photos and more than a thousand distinct classifications.

With keras.applications.Xceptionmodule in Python, leveraging this model in our code is quite simple. There will be three sections to the structure. The first tool is utilized to shrink the characteristics from the beginning of 2048 to 256. We're going to employ a Dropout Layer. Each of the CNN and LSTM will add one of them. The extracted characteristics anticipated by this model will be used as input after the photographs have been pre-processed using the Xception model (minus the output layer).

The second one is Sequence Processor: This Embedding layer, followed by the LSTM layer, will handle the text input. FinallyDecoder: To make the final predictions, we'll combine the results from the previous two layers and apply a dense layer. An output vector with a fixed length is produced by the component extractor and grouping processor. A Dense layer processes these after merging them. 256 Dense layers are used to classify the image. The final layer will have an identical number of nodes as our vocabulary size.

There are different Hyperparameters that determine the output. They are Activation function, Optimizers, Dense layers, Dropout rate, Epochs etc. Rectified Linear Unit (Relu) is the Activation function. Loss is calculated by categorical cross entropy. Optimizers are required for deep learning projects to adjust the features for a model. Adam is the optimizer used. The dropout rate is 0.5 which means 50 percent of the nodes to dropout. One loop across the entire training dataset is referred as an epoch. The epoch size is 10 in this project.

We will use 6000 photos with 2048-long feature vectors each to train our model. We will utilize a Data Generator because it is impossible to store all of this data in memory at once. We can construct data batches this way, which will speed up the process. In addition, we'll specify the no. of epochs, or cycles of training dataset, that the prototype must go through. The choice of this value must ensure that our model is neither overfitted nor under fitted. We'll utilize the model.fit generator() method. And depending on the CPU, the entire operation will take some time (Fig. 6).

In addition to the tokenizer file we made earlier, the sequence generator will now be used. For the specific image under observation, the first stage of feature extraction will be carried out. The function is manually supplied the way of one of the leftover 2000 test photos. A dictionary or a list can be used to keep track of the predictions for each image as you iterate through the test data set. In order to produce coherent phrases, the
<figure><img src="image_9.png" /><figcaption>Fig. 6. Neural Network structure</figcaption></figure>

real workings of picture formation use the beginning sequence, the end grouping, and recursively calling the framework.

# 5 Results

## 5.1 Qualitative Results

We utilise BLEU to gauge how closely the captions produced by our approach resemble those created by people. Bilingual Evaluation Understudy is what BLEU stands for. It is an algorithm that has been used to gauge the effectiveness of machine translations. A well-liked machine translation statistic called BLEU examines how often n-grams appear together in candidate and reference statements. Larger n-gram scores reflect fluency, whereas unigram scores (B-1) reflect the accuracy of the translation. It is simple to comprehend and calculate. BLEU is a multilingual system. It is located between [0, 1]. The greater the score, the higher the capping quality (Figs. 7, 8 and 9).

Only three photographs were tested for simplicity's sake, and the outcomes are shown in the following pictures (Table 2):

## 5.2 Quantitative Results

From the above table the Xception model has less error compared to other CNN models according to my project output (Table 3).
<figure><img src="image_10.png" /><figcaption>/content/drive/MyDrive/ML/testing酵母菌 generator.py' -i '/cont
2022-12-02 16:56:26.286060: W tf/1/1 [==================================================] - 3s 3s/step

start man is climbing up rock end</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>/content/drive/MyDrive/ML/testing酵母菌 generator.py' -i '/cont
2022-12-02 16:55:44.724048: W tf
1/1 [==================================================] - 3s 3s/step

start dog is running through the snow end</figcaption></figure>

<caption>Fig. 8. Caption for Image 2</caption>
<figure><img src="image_12.png" /><figcaption>nt/drive/MyDrive/ML/testing酵母菌基因组测序</figcaption></figure>

2022-12-02 16:54:49.267672: W tensorflow/core/common_runtime/gpu/
1/1 [==================================================] - 3s 3s/step

start man is walking on snowy peak end

<caption>Fig. 9. Caption for Image 3</caption>

<table><caption>**Table 2.** Comparison between predicted and original values</caption>
  <thead>
    <tr>
      <th>Image</th>
      <th>Predicted Description</th>
      <th>Original Description</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td></td>
      <td>Man is climbing up rock</td>
      <td>A man climb a rocky wall</td>
    </tr>
    <tr>
      <td></td>
      <td>Dog is running through the snow</td>
      <td>A dog run through snow</td>
    </tr>
    <tr>
      <td></td>
      <td>Man is walking on snowy peak</td>
      <td>A skier look at framed picture in the snow</td>
    </tr>
  </tbody>
</table>
<table><caption>**Table 3.** Experimental errors for different CNN models</caption><thead><tr><td>Model</td><td>Epochs</td><td>Error(in percentage)</td></tr></thead><tbody><tr><td>VGG-16</td><td>10</td><td>3.2058</td></tr><tr><td>ResNet</td><td>10</td><td>3.0749</td></tr><tr><td>Xception</td><td>10</td><td>2.7899</td></tr></tbody></table>

# 6 Conclusion and Future Work

With the aim of not only describing the immediate environment, but also assisting persons who are visually impaired in understanding their surroundings, A deep learning framework is introduced that instantly produce image captions. We can see from the data that the profound learning technology utilized here produced fruitful outcomes. Together, the CNN and LSTM had the option to decide the connection between objects in pictures by synchronizing their operations. Our model is not flawless and occasionally produces inaccurate captions. Currently, we are choosing the word with the highest probability as the next word in the sequence using a greedy method. Instead, beam search chooses a set of terms with the greatest likelihood and conducts simultaneous searches over every sequence. We could improve our prediction accuracy by using this strategy.

# References

1. Panicker, M.J., Upadhyay, V., Sethi, G., Mathur, V.: Image caption generator. In: Proceedings of International Journal of Innovative Technology and Exploring Engineering (IJITEE) ISSN: 2278-3075 (Online), vol. 10, no. 3, January 2021

2. Sharma, G., Kalen, P., Malde, N., Nair, A., Parkar, S.: Visual image caption generator using deep learning. In: Proceedings of 2nd International Conference on Advances in Science & Technology (ICAST) (2019)

3. Krishnakumar, B., Kousalya, K., Gokul, S., Karthikeyan, R., Kaviyarasu, D.: Image caption generator using deep learning. In: Proceedings of International Journal of Advanced Science and Technology (2020)

4. Vani, M.: Image to caption generator. In: Proceedings of International Journal for Research in Applied Science & Engineering Technology (IJRASET) ISSN: 23219653; IC Value: 45.98; SJ Impact Factor: 7.538, vol. 10, no. XI, November 2022

5. Wang, H., Zhang, Y., Xiao, S.: An overview of image caption generation methods. Comput. Intell. Neurosci. January 2020

6. Staniute, R., Sesok, D.: A systematic literature review on image captioning. Appl. Sci. 9(10), 16 (2019)

7. Zakir Hossain, M.D., Sohel, F., Shiratuddin, M.F., Laga, H.: A comprehensive survey of deep learning for image captioning. ACM Comput. Surv. 51(6), 1–36 (2019)

8. Li, J., Xu, N., Nie, W., Zhang, S.: Image captioning with multi-level similarity-guided semantic matching. Visual Informatics (2021)

9. Chaithra, V., Rao, C., Deeksha, Shreya, Jagadisha, N.: Image caption generator using deep learning. In: Proceedings of International Research Journal of Engineering and Technology, vol. 09, no. 01, January 2022
10. Sreejith, S.P., Vijayakumar, A.: Image caption generator using deep machine learning. In: Proceedings of International Journal of Trend in Science Research and Development, vol. 5, no. 4 (2021)

11. Khant, P., Deshmukh, V., Kude, A., Kiraula, P.: Image caption generator using CNN-LSTM. In: Proceedings of International Research Journal of Engineering and Technology, vol. 08, no. 07, July 2021

12. Subash, R., Jebakumar, R., Kamdar, Y., Bhatt, N.: Automatic image captioning using convolutional neural networks and LSTM. In: Proceedings of International Conference on Physics and Photonics Processes in Nano Sciences (2019)

13. Maroju, A., Doma, S.S., Chandarlapati, L.: Image caption generating deep learning model. In: Proceedings of International Journal of Engineering Research and Technology, vol. 10, no. 09 (2021)

14. Park, C.C., Kim, B., Kim, G.: Towards personalized image captioning via multimodal memory networks. In: Proceedings of IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 99 (2018)

15. https://github.com/jbrownlee/Datasets/releases/download/Flickr8k/Flickr8kDataset.zip

16. Ramprasath, M., Vijay Anand, M., Hariharan, S.: Image classification using convolutional neural networks. Int. J. Pure Appl. Math. 119(17), 1307–1319 (2018)

17. Yuvaraj, D., Hariharan, S.: Efficient content based image retrieval technique using multiple attributes. In: International Conference on Emerging Research in Computing, Information, Communication and Applications (ERCICA 2013), Nitte Meenakhsi Institute of Technology, Bangalore, 02–03 Aug 2013, India (2013)