Natural Disaster Tweets Classification Using Multimodal Data
Authors: Mohammad Abdul Basit
Bashir Alam
Zubaida Fatima
Salman Shaikh
Venue: EMNLP 2023 Main (2023)
Keywords: Multimodal Data
Multi task learning
NLP of Social Media data
AI for social good

Abstract: Social media platforms are extensively used for expressing opinions or conveying information. The information available on such platforms can be used for various humanitarian and disaster-related tasks as distributing messages in different formats through social media is quick and easy. Often this useful information during disaster events goes to waste as efficient systems don’t exist which can turn these unstructured data into meaningful format which can ultimately assist aid agencies. In disaster identification and assessment, information available is naturally multimodal, however, most existing work has been solely focused on single modalities e.g. images or texts separately. When information from different modalities are integrated , it produces significantly better results. In this paper, we have explored different models which can lead to the development of a system that deals with multimodal datasets and can perform sequential hierarchical classification. Specifically, we aim to find the damage and its severity along with classifying the data into humanitarian categories. The different stages in the hierarchical classification have had their respective models selected by researching with many different modality specific models and approaches of multimodal classification including multi task learning.  The hierarchical model can give results at different abstraction levels according to the use cases. Through extensive quantitative and qualitative analysis, we show how our system is effective in classifying the multimodal tweets along with an excellent computational efficiency and assessment performance. 
With the help of our approach, we aim to support disaster management through identification of situations involving humanitarian tragedies and aid in assessing the severity and type of damage.

Natural Disaster Tweets Classification Using Multimodal Data
Mohammad Abdul Basit', Salman Ghufran Shaikh', Bashir Alam' and Zubaida Fatima
'Department of Computer Engineering, Jamia Millia Islamia, India 2 KKing Abdullah University of Science and Technology, Saudi Arabia
3 Department of Electronics and Communication Engincering, IITT-elli, India salman. shaikh@kaust. edu. zubaidafh@gmaill com
mohammadbasi t3@gmail. com balam20 jmi ac. in
Abstract
dent upon a number of factors, these could be lives lost, damage caused, economic loss, etc. Each year, these disasters take place in the form of carth- quakes, floods, tornadoes and severe storms, hurri- canes and tropical storms, droughts and wildfires and cause plenty of damage to the human soci- ety. The impact due to these disasters cannot be neglected and thus first-responders such the lo- cal residents, health professionals and emergency workers must come to the aid of the damaged sites and affected individuals. The immediate priority that follows after a disaster occurs providing emergency aid to the injured people. However, it is also very important to assess the damage that has been caused to different structures and zones as that is good indication of the severity of the situation.
Social media platforms extensively used for expressing opinions or conveying informa- tion. The information available on such plat- forms can be used for various humanitarian and disaster-related tasks as distributing mes- sages in different formats through social media is quick and easy. Often this useful informa- tion during disaster events goes to waste as efficient systems don't exist which can turn these unstructured data into meaningful format which can ultimately assist agencies. In disaster identification and assessment, informa- tion available is naturally multimodal, however, most existing work has been solely focused on single modalities e.g. images texts separately. When information from different modalities are integrated, it produces significantly better re- sults. In this paper. we have explored different models which can lead to the development of system that deals with multimodal datasets and can perform sequential hierarchical classi- fication. Specifically, we aim to find the dam- age and its severity along with classifying the data into humanitarian categories. The different stages in the hierarchical classification have had their respective models selected by researching with many different modality specific models and approaches of multimodal classification in- cluding multi task learning. The hierarchical model can give results different abstraction levels according to the cases. Through ex- tensive quantitative and qualitative analysis, we show how our system effective classifying the multimodal tweets along with an excellent computational efficiency and assessment per- formance. With the help of our approach, we aim to support disaster management through identification of situations involving humanitar- ian tragedies and aid in assessing the severity and type of damage.
In times of disaster calamity, the information available on social media could be widely used in disaster assessment, management and analysis and for other humanitarian tasks. The data regarding these disasters that are collected using field surveys are often not available immediately. Disaster dam- age data such as the location, area or the zone and extent of the damaged structures and facilities is critical in disaster management operations (Hao and Wang, 2020). Therefore, becomes necessary for the first-responders of humanitarian aid to rely on data derived from social media, as many users post messages and information in different formats, i.e image, text, audio, etc. However, the utilisation of this data is not easy. One of the challenges that needs to be tackled before the data can be utilised appropriately is related with isolation of content useful for crisis management. In times of cri- sis, it is extremely important to determine which areas have been affected the most (hence require the most attention and should be an immediate pri- ority) and other humanitarian tasks that include missing/found people, injured or dead people and other sub-tasks. However, this becomes difficult to
1 Introduction
Natural disasters and calamities have long been occurring. The gravity these disasters is depen-

---

identify during these times, there will be huge volume of data that could mask the severity level of certain areas and people need. Therefore, there is a need for automatic systems to assess, identify and analyse the damage that has been caused which in turn could benefit the emergency management process notably (Agarwal et al., 2020). Although the extraction of information from social media re- sources to help humanitarian aid workers has been going on for some time, this has been limited to uni- modal data. While these unimodal damage analysis frameworks are efficient, they are not able to assess damage as effectively for social media posts (Agar- wal et al., 2020) that come in a multimodal format, whether it be text with images, text with audios, au- dios with images, etc. Thus, we intend to develop effective system that can leverage multimodal data and is able to assess, identify and analyse the severity of the damage caused and identify other humanitarian sub-tasks real time. The unimodal text feature is often not very helpful in disaster situations as many times the text of such tweets is incomplete or incomprehensible and the main idea is often conveyed by the images, leading us to choose the text and image modalities, which can give more information about event (Ofli al., 2020). The main reason for using multimodal data sources is that possible extract complemen- tary and richer information coming from multiple sensors, which can provide much more optimistic results than single input. Some monomodal learn- ing systems have significantly increased their ro- bustness and accuracy, but in many use cases, there are shortcomings in terms of the universality of dif- ferent feature levels and inaccuracies due to noise and missing concepts (Bayoudh al., 2022). Even though the approach beneficial. in practice is very challenging (Atrey et al., 2010) due to the dif- ferent noise and conflicts in the different modality data (Wang et al., 2007). Therefore one has to find a model which can balance the same. We explore different models both the modalities and select the one which best balances all the noise present in the different modalities. Further the design of the system should be such that it responsive, fast and efficient in its computation as the response is needed quickly during a disaster. The system should also have scope for addition and deletion of different categories which is only possible if the architecture is modular one and as a result choose the hierarchical structure. The system
should also be able handle unimodal tweets (text only). We leverage the recent advances in classifi- cation for images and text to build an efficacious system which can give much fine grained results.
2 Related Work
In the context of unimodal classification, Madichetty and Sridevi (2019) classified textual tweets from CrisisMMD (Alam et al., 2018) dataset into informative and non-informative categories using CNN and ANN. Alam et al. (2019) develop an automatic data processing service which takes in textual data from various sources and classify it into disaster type, informative and humanitarian information conveyed. For the purpose of classifi- cation, deep learning and classical algorithms are used. While these unimodal approaches produce commendable results, they have been surpassed by the multimodal approaches. Hao and Wang (2020) propose a data-driven method to locate and assess disaster damage with humongous multimodal social media data. The images have been classified using machine learning while the text follows a keyword search based method. Gautam et al. (2019) analyse the multimodal data related to different natural disasters, for instance: floods, earthquakes, etc. They propose novel decision diffusion technique to classify them into informative non-informative categories. Their approach of training an image and text classifier and combining the two outperforms the baselines. Zou et al. (2021) proposed a method through which they integrated image and text information to identify disaster images collected from different social media platforms. They use a deep learning method and FastText framework to extract visual and textual features respectively. They then develop data fusion model to combine these features and experiment on real world disaster dataset through the CrisisMMD dataset. Abavisani et al. (2020) present a new multimodal fusion method that uses both images and texts as input. They introduce a cross-attention module that is able to filter out uninformative and misleading components. Agarwal et al. (2020) present a damage identification and its severity detection system, called Crisis-Dias. Through qualitative, quantitative and theoretical analysis on a real- world multimodal dataset, they are able show that the information once presented together, often produces high-end analysis about the domain and

---

facilitates better learning performance. Mouzannar al. (2018) propose a multimodal deep learning framework to identify damage related information. their approach, they combine multiple pretrained unimodal CNNs that extract features from the raw texts and images respectively. Finally, a resultant classifier labels the posts based on both modalities. Their results on home-grown database of labelled social media posts shows good results and validates the use of the proposed method. Nalluru combination of semantic textual features with the image features to be able to efficiently classify a relevant multimodal social media post. They utilise feature generate framework which uses TF-IDF vectors and glove embeddings for the textual data, and pre-trained residual networks for generating vectors for the image data, the vectors extracted are then concatenated through which they build a LightGBM model to classify different disaster-events as informative or not. Their results demonstrate that features based on a hybrid framework (using both text and images) improve the performance of identifying relevant posts.
3.1 Task
The objective of this stage is to distinguish between Informative and Non-Informative samples. Any image or text which contains useful events which points to any of the signs of damage, suffering during a disaster has been labelled as Informative.
The dataset has separate labels for the image and text. For all tweets (4, i) where & is the text and i is the image, we formulate a binary function F in fonformaiive The data has separate labels for both image and text.
(2019) experiment with
Class Distribution:
Text: Informative: 11509 -Non Informative: 4549
Image: Informative: 9374 Non Informative: 8708
3.2 Task
The CrisisMMD dataset labels the tweets into further sub classes depending on whether they signify structural damage or people being affected in any way. The classes are so chosen that they offer critical information first respondents.
Knowledge of tweets conveying structural and humanitarian rescue information would be of great assistance to relief workers. To accomplish this, the classes from the original dataset have been aggregated into main classes for the system. The classes Affected Individuals, Injured or dead people, Missing or found people, Rescue Volunteering donation effort have been grouped into Humanitarian, the Infrastructure and Utility Damage, Vehicle Damage into Structure and Not humanitarian and Other relevant Information into Non Informative. The textual tweets have been enriched with tweets from the HumAID (Firoj Alam, 2021) dataset to reduce the class imbalance for the structure and humanitarian classes. The dataset contains text only tweets from natural disasters between 2016-2019 and labelled under similar labels as the CrisisMMD, as a result they increase the diversity of the dataset along with reducing the class imbalance. We formulate a tri-valued function Fhuman-struct for a given tweet M 1) where is the image and / is the text.
3 Dataset and Problem Definition
The dataset that was used for this project was the CrisisMMD dataset. It's labels were aggregated according the specific task at hand. The dataset included information that could be filtered exten- sively depending on the scenario. The CrisisMMD dataset consists thousands of annotated tweets and images collected during seven various natural disasters including earthquakes, hurricanes, wild- fires, and floods that happened in the year 2017 across different parts of the world. The data is hi- erarchical the class labels at each stage depend the annotation in the previous stage. The above data is further complemented by text only dataset called HumAlD2 for certain downstream tasks. We use the several sub-categories of the classes to pro- duce our desired results which to be able to iden- tify the structure damaged and it's severity along with humanitarian mishaps. To do this, based on the dataset we formulate the following tasks.
The purpose of aggregating the sub classes into more generic classes at this level is so that the number of classes classify remain small along with events having similar features being clubbed into a single category which can later undergo finer classification into their respective sub classes. We include the Non Informative class at this stage and
hnttps://crisisnll.. http:/pppgrr...ogrriiiissseeseee httpa/////rrsssslp.ggri..rrr datttat

---

the upcoming ones since it hierarchical system, any samples earlier miss-classified can be removed at further stages thus reducing the rate of errors. As a result these samples will not be processed any further by the system as they don't contain any useful information. As the above task, the image and text have different labels.
people, missing or found people, rescue needed as rescue volunteering or donation effort and all other classes are placed in no human. The original dataset has different annotation for image and text but we have assigned a single class for a text-image pair by following the following approach: If either of the tags contain class from people affected, the entire pair is people affected, rescue needed follows next and if none present falls into human. The people affected class has been given more priority over the rescue needed as often peo- ple injured, in critical conditions need medical care quickly and have to be attended to by paramedics urgently.
Class Distribution
Text (After enrichment from HumAID): Hu- manitarian: 9572 Non Informative: 9359
Bruuture(Damagee 8677
Image: Humanitarian: 2917 -Non Informative:
11237- Structure(Danage): 3928
3.3
Task 3
Class Distribution:
The tweets from the Strmature(Damaged) in the pre- vious task come over to this stage. The objective of this stage is to be able to determine the spe- cific structure which severity, which would assist the rescue ops to tar- get their efforts in a more targeted and prioritised manner. The tweets which belonged to the struc- ture class have been manually annotated to identify the damaged structure into one of the following classes: buildings, roads/bridges/vehiclee, and no structure and the severity into one of the follow- ing: Severe, Mild and No damage. The structure identification and subsequent severity tagging was done mainly for tweets which contained images with Structure class as the text information is often not enough to estimate the severity of a damaged structure. The no structure and No damage classes have been included keeping in mind the plausible miss-classifications in earlier stages. The task as- signs two labels to every such tweet to indicate the damaged structure and the severity. The distribu- tion of the tweets is such:
People Affected: 1648 Rescue Needed: 4502 No Human: 11932
damaged along with its
Methodology
4.1
Text Preprocessing
Lower case the text: We lowercase all the text so that it is in the same case.
Remove URLs, Mentions (@), RTs: We move all URLs starting with http/https, men- tions which start with @ sign, and any RT signs i.e. Retweeted as they are not useful for text classification.
Remove unnecessary characters: Removal of all the extra spaces, operator signs, Non- ASCII characters, punctuation marks and sin- gle characters this solely increases the length of the sample without adding much information.
Structure Damaged: Buildings: 1886 Roads- bridges-vehicles: 646 No-Structure: 13547
Add space between words and removing extra spaces: The space has to be inserted between the words whilst making sure there extra spaces between two words.
Damage Severity: Severe: 2446 Mild: 895 No
Damage: 14741
3.4 Task 4
4.2 Image Preprocessing
The tweets which are marked as humanitarian in Task 2 make their way to this task. This task aims at further sub classifying the humanitarian tweet to the specific message it conveys. The classes in the original dataset have been combined into more generic classes as there existed multiple classes which conveyed similar information. The classes that we classify into are people affected, which an aggregate of affected individuals, injured or dead
We apply the following augmentation on the im- ages so that the model generalises well over data which it hasn't seen and prevents overfitting of data to the training samples. We perform randomised setting of the following characteristics of the image: flipping, rotation, zoom, height, width. All values are transformed in the range of to along with resizing of the image to the models requirements.

---

4.3
Task 1
so that the individual noise present in the differ- modalities doesn't degrade the performance of the classifier. For this we are using the Projection layer. To handle class imbalance in the data we use class weights and sample weights by assigning more importance to minority class samples. The reason we are experimenting with multi task learn- ing is because the two tasks are similar, they share the same representation and there is common information between the two.
The function for this stage is formulated as simple OR operation.
Liiformativee
u = oor 8
formanas (1
(toon infommailoma S. 0 and = where 3g represents the output of the model for modality x.
The rationale behind using an OR operation is that often images could convey life saving information which the text might miss out and vice versa. For this purpose separate classifiers are trained for both the text and image modalities and the outputs of their classification are combined with an OR operation. The imbalance in textual tweets have been overcome using the SMOTE (Chawla et al., 2002) algorithm. The non informative tweets are dropped and the remaining propagate further down the pipeline.
4.6 Task
The tweets which are marked as humanitarian in Task make their way to this task. This task fol- lows similar approach as the second approach in Task 3, i.e. to train classifier by combining the image and text feature vectors into one. The class imbalance is handled by class weights.
5 Experimental Setup
4.4 Task 2
For this stage, we train 2 separate classifiers for the image and the text data in similar fashion to Task 1, after which the results are fused using valued function F Fhuman-strud struct as below.
We experiment with various state of the art text embeddings and image embeddings utilising newer models like Image transformers in addition to al- ready successful CNN based models. The models have been fine tuned and while some of them have had partial training to fit our use-case. The mod- els are run for a maximum of 100 epochs but with carly stopping and a set patience value. The mod- els with best validation loss and recall are stored at every stage. The optimiser used in the respec- tive models vary between Adam (Kingma and Ba, 2017). AMSGrad (Reddi et al., 2018) and Stochas- tic Gradient Descent. The activation functions are either ReLU or Softmax. The Cross Entropy loss function is used which can be binary or categori- cal based on the number of classes. The tokeniser used for Bert and its variants are BERT tokeniser GPT tokeniser' for GPT2 and its variants and the Keras tokeniser for others. The projection layer in Task 3 takes as input the embedding produced by encoder and applics the Gaussian Error Lin- ear Unit (Hendrycks and Gimpel, 2016) activation function on it. It then passes it through dense layer of 256 neurons. The multiple vectors are then added using the Add layer and a single vec- tor is obtained. The output passes through layer
2(Suucture)
i = or or
Faumu--struattt
OHmmmnitrria
eli & o
Informative)
where 3g represents the output of the model for modality x.
The structure has been given more priority as it is imperative to figure out major structural damages which could otherwise obstruct relief operations. For text the imbalance is resolved using the HumAID dataset while for images after trying a variety of methods we have stuck with class weights which give more weightage to samples from minority classes.
4.5
Task 3
This deals with the structure/damage related tweets from stage 2. In this, we consider approaches. In the first we obtain a common feature vector for both the text and image modalities and use it to train a multi task classifier to predict damaged structure and severity. In the second approach we still use the combined feature vector but to only predict the damage severity, the structure damaged is classi- fied using a separate image classifier i.e. train 2 separate classifiers. Since we are using multi- ple feature vectors from multiple modalities we need to have a common representation the both
http:://wwnngniaaaeoooooo////ttr wwt/ooss/tansoormmeeeeeee/e model_ Meee_ddc//brrtt
"https: hhttt/////eiiiiiee....oooooooors httpr//nnaeeeeoooo////tattemeeee model_doc/gpt>
htttps: //kesss///kas...ae http:/pp///eeraaanl//oeeeeeeeeea
tokenizer/

---

Task
Model
Precision
Recall
F1-Score
Acc.
6
Results
Word2Vec
0.80
0.78
0.79
0.83
BERT
0.81
0.79
0.80
0.83
Taskl Text
For Task I, we have selected BERT (Devlin et al., 2018) for the text pipeline as it outperforms all the tion (77%) as ResNet (80%) has a higher recall for the informative class and our main aim is to reduce the false negatives (we don't 't want the infor- mative tweets to be labelled as non informative), Upon performing an OR operation the results of the selected image and text pipelines see that the accuracy after using data from both modalities has increased to 86%. Which higher than any of the accuracy of the single modalities. Also the recall of the informative class has also increased to 94%, again more than both the unimodal recalls, ALBERT (Lan et al., 2019) has the higher values for all the metrics and since it is light weight we will be choosing that for our text classification in Task 2. It also has good recall for the classes that are considered the positive ones. For Image, We see that the DelT (Touvron et al., 2021) has ob- tained higher recall and Precision than all the other models. As a result we also see how the vision transformers are better at image classification task. We see that after combination of the prediction of modalities, the accuracy has not increased but the quality of the predictions has. The structure class had pretty low metrics for ALBERT (recall: 72%) and the DelT model had low recall for the humani- tarian class (72%), but upon combination there a balancing effect and the results of all the 3 classes have had a boost. Therefore it is somewhat of an ensemble effect. The recall of the damage class has increased as a result of adding the Image model and the humanitarian class has seen an increase in the recall due the addition of the text model. After testing the 2 approaches in Task 3, we coneur that MultiTask DeIT-ALBERS has better metrics. While naturally the single output models should have had better results, the deviation is due to the noise which could be present in a certain target output in the training samples as result the model could have been biased towards something. Where as in the Multimodal the model has concurrently optimise the loss function of the two outputs as a result it can cancel the noise which was present in one target specific sample and absent in the other. For Task 4, the performance of Inception and DelT
FastText
0.82
0.77
0.79
0.83
Glove
0.79
0.78
0.79
0.82
XL.Net
0.81
0.76
0.78
0.83
others. For Image can clearly see that Xcep- tion (Chollet, 2017) performs better than all the models followed by ResNet (He al., 2015). But despite the higher results we will not choose Xcep-
DefT
0.81
0.81
0.81
0.81
DenseNet
0.79
0.79
0.79
0.79
Task1
Xception
0.82
0.82
0.82
0.82
Image
ResNet
0.82
0.81
0.81
0.81
Inception
0.81
0.80
0.80
0.80
V3
BERT-
Multi- modal
0.83
0.77
0.79
0.86
ResNet
ALBERT
0.77
0.81
0.79
0.85
BERT
0.75
0.80
0.77
0.84
Task2
DistiBERTT
0.74
0.79
0.76
0.83
Text
DistilGPT-
0.74
0,80
0.76
0.83
2
XLNet
0.73
0.80
0.76
0.82
Xception
0.70
0.76
0.72
0.75
ResNet
0.73
0.78
0.75
0.79
Task2
Inception
0.77
0.80
0.78
0.83
Image V3
BiT
0.76
0.80
0.78
0.82
CalT
0.76
0.81
0.78
0.82
DeIT
0.80
0.82
0.81
0.85
ALBERT-
Multi- modal
0.80
0.82
0.80
0.81
DelT
Table 1: Results of Task and Task 2
normalisation and the vector can be used as a fea- ture vector. The models and their Fl score, Recall, Accuracy and Precision are reported and the best model is selected based on the accuracy and recall for the informative class. The learning rates for the models are mostly 0.001 except for certain cases where is 0.01 or 1e-5, The experiments have been performed on a NVIDIA TESLA P100 GPU.
Precision
Fi.Score
Acc.
Multitak Damage
0.83
0.88
0.86
092
ResNet-
Seructure
0.73
0.85
0.78
0.91
DERT
Multitask
Damage 0,84
692
DelT-
Structure
0,80
0.88
0.83
0.93
ALBERT
HERT-
Damage
0.78
0.81
0.79
0.90
Inception
BERT.
Damage
0.80
0.85
0.82
0.90
DefT
VGG11
Stnacture
0.76
0.89
ResNett152
Stracture
0.79
0.84
0.81
0.92
Inception-
Human-
Task4
BERT
Marian
DETT-
Himan-
0.85
0.88
Table 2: Results of Task 3 and Task 4

---

Text
Harmer
Figure I: Final System Diagram
is almost comparable but the DelT has a higher computation cost, for this specific task the DelT took times the time taken by Inception do the same task. Since time of the essence in our application, we have chosen the Inception model (Szegedy et al., 2016) for this task.
image and inclusion of text will not help in most samples. The weightage given to the text and image embeddings are same and both contribute equally in the prediction of the damage severity. In Task 4, the text is fed to the BERT encoder and image to Inception, the output of these encoders are concatenated to form the feature vector. The models used in Task 4 are pretrained.
7 Final System Discussion
Based on our experiments we have selected the components as discussed above and the detailed final system is shown in Figure 1.
To analyse the effectiveness of the system, we calculate the Fl-score and accuracy of the system as the average of the 4 tasks, the F1-score and accuracy comes and 88% respectively, which is quite impressive. On top of having an accurate classification, the system also has to be responsive and provide quick predictions as it will be used in disaster sites which are time critical. The system is tested on GPU as well as an older machine on CPU to see how well it performs in constrained environments. To estimate the time that model takes in inference we run around 100 tweets and calculate the time a tweet takes to be processed per Task. The inference was run separately on a Intel Core 6th generation CPU with 2.6GHz speed (an old CPU) and NVIDIA Tesla P100 (GPU).
Performance:
Design: The embeddings from BERT used for text in the Task are summation of the last four layers, which is then input to GRU layer. The ResNet 152 model in the image pipeline has all its layers unfrozen except the batch normalisation layers, which are kept frozen so that the learnings of the ImageNet dataset are preserved. The informative tweets progress to Task 2 and the others dropped at this stage. At stage 2, the feature vector obtained by concatenating the output of the last layers of the ALBERT model, which then fed to the LSTM layer. The DelT used for Image a pretrained model which used to encode the images and then classified using an ANN of size 512 neurons. The outputs are fused using the Fiuman struct- The Structure/Damage related tweets reach the Task 3. In Task 3 Multi Task model where the text features are obtained from ALBERT and DelT used for the Images. The reason for taking only the image projections for structure identification is that the structure is usually conveyed from the
be 83%
Tiotl = Tiaskl + Tiask2 + maar(Tusks. ((3)k4)(3) We find on average tweet takes 400 milliseconds the GPU and 1.32 seconds on the CPU over the entire process. The model boasts impressive speed on GPU well as decent processing speeds the absence of complex architectures as evident from the old CPU's processing time, thus showcasing the resilience of the system.

---

Task
Model
Precision
Recall
Accuracy
F1-Score
Multi Modal Logistic Regression Decisson Policy (Gautam et 2019)
Task
Unimodal Text Piplcine (Madichetty and Sridevi, 2019)
76
76
75.9
76
Multi-modal BERT RERT-ResNet (Ours)
83
77
SES Cross. BERT DenseNet (Ahavisani et al., 2020)
72.65
59.76
Tusk (Dumage Class)
Multi Task DeUT-ALBERT (Ours)
84
88
MaltiMdaal CNN VGG16 (Olli al., 20200
78.3
78
78.4
78.3
Tisk
Cross-BERT-Deaee (Abavisami al., 2020)
91,14
68.41
Inception-ERRI (Owrs)
Table 3: Comparison with similar studies
Example-1:
Even if non informative tweets enter a stage, due to the hierarchical nature they can be removed every stage involves non-informative class. The data can be viewed at any granularity level by only allowing tweets go to a particular stage. Further more if newer and more accurate models are present for the tasks at any stage they can be easily incorporated in the system owing to its modular design, giving it plug and play feature. The variety of fine grained classes in the system allow the rescue workers to formulate priority list based on a combination of classes and respond the emergencies in swift manner. Along with having great performance for multimodal tweets, the system can generalise well to unimodal tweets as well, as the first tasks are a simple fusion and the absence of a modality doesn't interfere with the others' classification. For the last 2 phases, the vector of the missing modality can be left blank, as result we can adapt to variety of tweets without considerable decrease in quality.
Features:
Figure 2: Tweet Text: California wildfire evacuees just want to "go home" if they have a home still standing
http::/h.colwOxkkyyiitt
Actual tag: Task I: informative; Task 2: hu-
man; Task 3: affected individual
Predicted: Task 1: Text: informative - Image:
informative; Task 2: Text: non-informative Image:
human; Task 3: affected individual
What we see in Figure 2 is that the classifier pre- dicts all the labels for the image accurately. Even though the text was non informative in the second task, the overall sample is classified as human class due to the Fruman struct operation of the predic- tions of the individual modalities.
Comparison: While comparing with similar work in this field, there doesnt exist exact exper- iments on the classes the sub divisions which we divide our data into. Though there does exist papers which deal with subset of our tasks and we can do somewhat of task by task comparision to evaluate our system with existing state of the art. It can be observed from the results present in Table 3 that our system outperforms models from similar studies for certain subtasks and produces fine grained classifications with greater flexibility by adopting modular structure.
Example-2:
It observed in Figure 3 that the model correctly predicts the tag and the benefit of the 2 modalities is again highlighted when the text and image give their different predictions in Task but we have been able to preserve the important content present in the photo.
Qualitative analysis: We perform qualitative analysis of few images to show how the model predictions work. This also justifies the design choices which we have made in terms of the model selection.
8 Conclusion
Through our rescarch we have been able to propose a hierarchical model which classifies the given mul- timodal tweet into the different categories of either humanitarian or structural damage. A thorough

---

Limitations
The quality and variety of the data used to train the models varies as no incident can capture all the possible combinations. There was a certain possibility of data being biased and ultimately de- veloping an unfair model. We actively strived to identify and mitigate potential biases in the train- ing data. The system currently works with tweets in English language and further study has to be performed to scale it to more diverse set of lan- guages. The support for unimodal tweets in Task 3 and Task 4 could be improved. Often tweets from humanitarian categories could represent damage categories also, which could be confused by the system. The system depends on the availability of internet connectivity with the masses which could not always be available if the network infrastruc- ture is itself damaged and limits the deployment of the system. Also the system relies on data available on networking sites and expects that the data will always be the truth. Determination of futh-fullness is currently not available.
HOTER
Figure 3: 'weet Text: The Hotel Heroes of Hurricane https:/hcc//oAAAwbr
Irma http::/tcco/Ge3FFiCC..
Actual tag: Task 1:
informative; Task 2: dam- age: Task 3: Severity: severe- Structure: building
Predicted: Task I: Text and Image: informative Task 2: Text- non informative: Image- damage; Task Severity: severe Structure: building
analysis is performed on a combination of Crisis- MMD, HumAID datasets for the purpose of the development of the model. It is also very respon- sive and can be fitted with different tasks as and when data for such disaster events is available. As a future work on this task, the speed of predictions could be improved gigantically by deploying the models on cloud computing platforms with GPU support and multiple instances with auto scaling. The system could implement zero shot learning as the photos from a natural disaster are not from a closed domain and a model will always not see events similar to what it was trained on. Currently the model has been tested for tweets from Twitter, but similar platforms like Instagram, Facebook, etc. could also be used with this model, we would need develop an adapter for the particular platform specifying rules for how the image and/or text is obtained from the platform. A future step would be to increase the diversity of the dataset by including samples from myriad social networks. We plan to expand the type of modality to include videos from platforms like YouTube and the like, precursor to which would be creation of a dataset for the modal- ity. Also the predictions could be further improved by taking user feedback on classifications and us- ing newer vision transformers as well as decision fusion techniques for combining the modalities.
Ethics Statement
The intended use of the system is areas struck with natural disasters and help the greater cause of disaster management. In this project, we used publicly available data, we did not use any data that was considered sensitive or private information that would require explicit consent. We remained com- mitted to stay on top of holding data privacy regu- lations and respecting individuals privacy rights in terms of data collection. The tweets are represented by tweet IDs which can be used to fetch the actual tweets. If user restricts or deletes his tweets, it cannot be fetched. Incase of development of a more detailed application which utilises the personally identifiable information, storage guidelines have to be followed. We remained focused constructive and positive applications, particularly aiding dis- aster response efforts and explicitly state that the AT model was not used for any illegal activities or harmful purposes.
References
Mahdi Abavisani, Liwei Wu, Shengli Hu, Joel Tetreault. and Alejandro Jaimes. 2020. Multimodal categoriza- tion of crisis events in social media. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

---

Mansi Agarwal, Maitree Leekha, Ramit Sawhney, and Rajiv Ratn Shah. 2020. Crisis-dias: Towards multi- modal damage analysis- deployment, challenges and assessment, Proceedings of the AAAI Conference on Artificial Intelligence, 34(01):346-353.
Haiyan Hao and Yan Wang. 2020. Leveraging multi- modal social media data for rapid disaster damage assessment. International Journal of Disaster Risk Reduction, 51:101760.
Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. 20155 Deep residual learning for image recogni- tion. CoRR, abs/ 15 12.003885.
F. Alam, M. Imran, and Ofli. 2019. Crisisdps: Crisis data processing services. Spain. Universitat Politec- nica de Valencia.
Dan Hendrycks and Kevin Gimpel. 2016. Bridging non- linearities and stochastic regularizers with gaussian error linear units. CoRR, abs/ 606.08415.
Firoj Alam, Tanvirul Alam, Md. Arid Hasan, Abul Hasnat, Muhammad Imran, and Ferda Ofti. 2022. MEDIC: a multi-task learning dataset for disaster image classification. Neural Computing and Appli- cations, 35(3):6609-26) 263-
Diederik P. Kingma and Jimmy Ba. 2017. Adam: A method for stochastic optimization.
Firoj Alam, Ferda Ofli, and Muhammad Imran. 2018. Crisismmd: Multimodal twitter datasets from natural disasters. Proceedings of the International AAAI Conference Web and Social Media, 12(1).
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. 2019. ALBERT: A lite BERT for self- supervised learning of language representations. CoRR, abs/1909. 11942.
Pradeep K. Atrey, M. Anwar Hossain, Abdulmotaleb EI Saddik, and Mohan S. Kankanhalli. 2010. Mul- timodal fusion for multimedia analysis: A survey. Multimedia Syst., 1656))34--379
Sreenivasulu Madichetty and M Sridevi. 2019. Detect- ing informative tweets during disaster using deep neu- ral networks. In 2019 11th International Conference Communication Systems Networks (COMSNETS), pages 713-
Khaled Bayoudh, Raja Knani, Fayçal Hamdaoui, and Abdellatif Mtibaa. 2022. A survey on deep mul- timodal learning for computer vision: Advances, trends, applications, and datasets. Vis. Comput.. 38(8):2939-299) 2970.
Hussein Mouzannar, Yara Rizk, and Mariette Awad. 2018. Damage identification in social media posts using multimodal deep learning. In ISCRAM.
Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. 2002. Smote: synthetic minority over-sampling technique. Journal of artifi- cial intelligence research, 16:321-357.
Ganesh Nalluru, Rabul Pandey, and Hemant Purohit. 2019. Relevancy classification of multimodal cial media streams for emergency services. In 2019 IEEE International Conference on Smart Computing (SMARTCOMP), pages 121-125.
Francois Chollet. 2017. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Ferda Ofli, Firoj Alam, and Muhammad Imran. 2020. Analysis of social media data using multimodal deep learning for disaster response. arXiv preprint arXiv:2004.1 1838.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and
Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language under- standing. CORR, abs/1810.04805.
Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. 2018. On the convergence of adam and beyond. In Interna- tional Conference on Learning Representations.
Muhammad Imran Ferda Offi Firoj Alam, Umair Qazi. 2021. Humaid: Human-annotate annotated disaster incidents data from twitter. In 15th International Conference on Web and Social Media (ICWSM).
Ramit Sawhney, Puneet Mathur, Ayush Mangal, Piyush Khanna, Rajiv Rain Shah, and Roger Zimmermann. 2020. Multimodal multi- task financial risk forecast- ing. In Proceedings of the 28th ACM International Conference on Multimedia, MM '20, page 456-465, New York, NY, USA. Association for Computing Machinery.
Muhammad Imran Michael Aupetit Firoj Alam, Ferda Offi. 2018. A twitter tale of three hurricanes: Harvey, irma, and maria. In Information Systems for Crisis Response and Management (ISCRAM) 2018.
Rajiv Ratn Shah. 2016. Multimodal analysis of user- generated content in support of social media applica- tions. In Proceedings of the 2016 ACM on Interna- tional Conference Multimedia Retrieval, ICMR '16, page 423-426, New York, NY, USA. Association for Computing Machinery.
Fahimeh Fooladgar and Shohreh Kasaei. 2019. Multi- modal attention-based fusion model for seman- tic segmentation of rgb-depth images. ArXiv, abs/1912.11691.
Akash Kumar Gautam, Luv Misra, Ajit Kumar, Kush Misra, Shashwat Aggarwal, and Rajiv Rain Shah. 2019. Multimodal analysis of disaster tweets. In 2019 IEEE Fifth International Conference on Multi- media Big Data (BigMM), pages 94-103.
Lin Sun. Jiquan Wang, Kai Zhang, Yindu Su, and Fang- sheng Weng. 2021. Rpbert: A text-image relation propagation-basee bert model for multimodal ner. Proceedings of the AAAI Conference on Artificial Intelligence, 13(155:13860-13888

---

Christian Szegedy, Vincent Vanhoucke, Sergey loffe. Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference Computer Vision and Paltern Recognition (CVPR).
Hugo Touvron, Matthieu Cord, Matthijs Douze, Fran- cisco Massa, Alexandre Sablayrolles, and Herve Je- gou. 2021. Training data-efficient image transform- ers amp; distillation through attention. In Interna- tional Conference on Machine Learning, volume 139, pages 10347-10357.
Shagun Uppal, Sarthak Bhagat, Devamanyu Hazarika, Navonil Majumder, Soujanya Poria, Roger Zimmer- mann, and Amir Zadeh. 2022. Multimodal research vision and language: review of current and emerging trends. Information Fusion, 17:149-171.
Surong Wang, Manoranjan Dash, Liang-Tien Chia, and Min Xu. 2007. Efficient sampling of training set large and noisy multimedia data. ACM Trans. Multimedia Comput. Commun. Appl., 34-5s14-es.
Zhiqiang Zou, Hongyu Gan, Qumying Huang, Tianhui Cai, and Kai Cao. 2021. Disaster image classifica- tion by fusing multimodal social media data. ISPRS International Journal of Geo- toooofformatin, 10(10).


Reviews:
[{"review_id": "R77K7aloLX", "reviewer": "Reviewer_iDm4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments, we were thrilled to see your interest in the paper.\n\nNonetheless, we would like to clarify on the balancing of dataset. The dataset which we aim to balance is the training data which would be used for the model's initial learning. The balanced dataset helps in achieving greater classification performance and efficient models as has been proven by multiple researches. \nThe data which would be fed to the model in a real world scenario would definitely be heavily skewed and that can be handled as the model would only be performing predictions (classification) on the data and wouldn't be used to train it on the go, thus preserving the performance.\n\nAlso overtime a balanced data set could be created from the predictions which could be used to further train the model to keep up with the recent trends.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Solid augmented dataset for the multi modal task\n- nice pipeline for putting together existing tools for a common goal\n- marginally improve performance on existing datasets", "reasons_to_reject": "- Balancing of dataset weakens the contribution. In a real world scenario the imbalance in Twitter data is considerable, rendering the findings not applicable in this scenario.", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this work, the authors present a multi-modal approach to classifying natural disaster tweets. The system incorporates both bert-like transformer models (for text) and deep learning models (for images). The paper is well written and sound, and the authors provide all needed details, code, and data. The evaluation is thorough and the enhanced dataset they produce is a solid contribution on its own. The methodologies used are not new, but the pipeline putting them together and accompanying analysis is a valuable enough contribution.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "B7vtBnzkMl", "reviewer": "Reviewer_SknH", "summary": "", "questions": "- For Task1, did you test a more restrictive combination with informative prediction if βt = 1 AND βi = 1?\n\n- In specific cases (e.g. DeIT), the F-score value could be greater than precision and recall. Could you confirm that is explained by the weighting of classes for F-score calculation?\n\n- The meaning and the use of formula (3) line 583 is not clear. Could you explain this?\n\n- The paper takes into account images and texts from tweets. Do you plan to use other kind of textual data (e.g. news) and other kinds of data (like video) in future work?\n\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad that you were impressed by our approach to divide the issue into tasks and the problem being solved. Though, to further ameliorate your confidence in our paper, we would like to address your concerns.\n\nWe agree with the view that it is indeed a sequential system as a step is preceded by some step, i.e something which is being tested for task 2 has to first be classified as informative by task 1 and so on. But the reason for calling it hierarchical is that their is a sort of tree like pattern in which a certain tweet is classified. For instance an informative tweet can be any of Humanitarian or Structure classes. Furthermore, a structure could be classified into further sub categories based on damage and severity, similarly the humanitarian  class is further sub divided into 3 classes. This hierarchy is what we meant by the title.\n\n**Answers to Questions to Author**\n1. Yes during the testing of the model for Task 1, we have tested all the four combinations, and specifically for the pair you have mentioned with \\\\(\\\\beta_i\\\\) and \\\\(\\\\beta_t\\\\) as 1 , we had a total of 1688 samples out of which 97% were classified correctly. The combination can work with these samples since it is an OR operation and the accuracy of correct prediction for such samples will be extremely high.\n\n2. Thanks for your observation, we have investigated the matter, and it is a result of human error. The values for F1 score and accuracy have been swapped for the **Task2 Image** and **Task4**. The issue missed the eyes of the authors despite multiple reviews. \nThe change is simple and we would need to swap the values for the 2 columns in the 2 sub categories. This would not have any effect on the final outcome and results (F1 score and Accuracy) as they have been calculated using the correct values for the two and this is just an error on our part while filling the tables.\nAfter correction the 2 parts of the table 1 and table 2 would look as follows:\n\n| Task     | Model | Precision | Recall | F1-Score | Acc.|\n| :------| :------: | ----: |-------|-------|----|\n|             | Xception   | 0.70 | 0.76 | 0.72 | 0.75 | \n| Task2 |   ResNet   | 0.73 | 0.78 | 0.75 | 0.79 |\n| Image |  Inception  V3 | 0.77 | 0.80 | 0.78 | 0.83 |\n|             |  BiT   | 0.76 | 0.80 | 0.78 | 0.82 |\n|             | CaIT | 0.76 | 0.81 | 0.78 | 0.82 |\n|             | **DeIT**| **0.80** | **0.82** | **0.81** | **0.85** |\n\n*Changes in Table 1*\n\n| Task     | Model |Class| Precision | Recall | F1-Score | Acc.|\n| :------| :------: | ---|----: |-------|-------|----|\n|             | **Inception-BERT**| **Humanitarian** | **0.88*** | **0.84** | **0.86** | **0.89** | \n| Task 4 |   DeIT-BERT | Humanitarian | 0.85 | 0.84 | 0.84 | 0.88 |\n\n *Mistakenly mentioned as 0.84 in anonymous submission paper\n\n\n*Changes in Table 2*\n\n\n3. The formula has been used to postulate the time it takes for 1 tweet on average to be predicted by the complete pipeline (go through all stages). The \\\\(T_{task}\\\\) specifies the time it takes for a tweet to go through the task. The task 1 and 2 are sequential so their times have been added up while task 3 and 4 could be done in parallel and therefore a max of their value is taken (A tweet can only go for one of the tasks). The time calculated in paper is when the model is run on an older CPU system to show the resilience of the system and how it performs decent even on a non GPU machine. When run on a GPU (ideally a multimodal system should run on a GPU), the model boasts a speed of *400ms/tweet*. \n\n\n4. Currently we had tested the model primarily for tweets from Twitter (now known as X), but definitely similar platforms like Instagram, Facebook , Weibo , etc could also be used with this model, we would need to develop  an adapter for the particular platform specifying rules for how the image and/or text is obtained from the platform. The adapter could be developed in this sense for any social media platform and it would be good to go with the model. As a future work, a dataset could be curated containing samples from a myriad of social networks and news websites and help in development of a more robust system. The model currently works with textual data, image and a combination of both as already mentioned in paper. Video is currently not supported but we plan on utilizing videos from youtube and similar platforms and integrating that as a modality in the system. A precursor to that would be the curation of a dataset for the same. We have currently not mentioned these in the future discussion but would be open to the edit. Furthermore, in terms of more effective model building, more experimentation could be done with newer vision transformers as well as decision fusion techniques for combining the modalities.\n\nRegarding Reproducibility, we have specified the parameters of the core model in the paper itself in the experiments and final system discussion headings along with a mention of setting for all the experiments and relevant code in the supplementary materials.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The motivations to exploit social media and multimodality in disaster assessment, management and analysis of the severity of the damage and humanitarian sub-tasks in real time are convincing.\n\n- The four subtasks addressed and methods associated with are clearly summarized.\n\n- Experiments are conducted with several state-of-the-art text and image embeddings.\n\n- Based on the results of experiments a general system (with a clear pipeline summarized in Figure 1) and relevant qualitative analysis are proposed and implemented.", "reasons_to_reject": "- Weak originality of the approaches used but an attractive combination of up-to-date methods well evaluated and integrated in a final system.\n\n- The 'hierarchical' concept proposed is not really clear. I am not sure that is a right term to use to describe the proposed method. In my opinion, the 'sequential' term could be more relevant.\n\n- The paper should extend future work discussion by considering other kind data. ", "questions_to_authors": "- For Task1, did you test a more restrictive combination with informative prediction if βt = 1 AND βi = 1?\n\n- In specific cases (e.g. DeIT), the F-score value could be greater than precision and recall. Could you confirm that is explained by the weighting of classes for F-score calculation?\n\n- The meaning and the use of formula (3) line 583 is not clear. Could you explain this?\n\n- The paper takes into account images and texts from tweets. Do you plan to use other kind of textual data (e.g. news) and other kinds of data (like video) in future work?\n\n", "paper_topic_and_main_contributions": "This papers deals with approaches based on multimodal datasets in order to propose sequential hierarchical classification. The hierarchical model can give results at different abstraction levels based on dedicated use cases. This paper highlights how the proposed system is effective in classifying multimodal tweets.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PZPRr96rNf", "reviewer": "Reviewer_j8SL", "summary": "", "questions": "The authors mentioned, in line 586, they have a reasonable time to execute the whole system, calculating on average 1.32 seconds per tweet. Honestly, the Twitter community produces a lot of tweets per second, and when we dealing with natural disasters quick response is crucial, so in my opinion, 1.32 seconds per tweet is a lot of time. So, be careful with this kind of conclusion or insight. \n\nSecond, the authors stated in line 572, they reached an overall f1-score from the whole system of 83%, which I agreed with it seems very impressive, but maybe it would be nice to compare it with other related work, I know that maybe there are no systems similar in the literature, but systems dealing with a subtask of your system yes, so it is important to locate your contribution into the state of the art.\nSo, I will appreciate some addition to this in the manuscript. \n\nMinor issues:\nFigure 1 is very low quality, it is very difficult to appreciate it. \nWhy used SMOTE in textual tweets and don't in the other kind of task? For instance, in task 4, you used class weights to deal with class imbalance.\nCould you give some ideas about why multimodal is not getting better results (with a large margin) versus text only?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your time and effort in providing us some great and insightful reviews for the paper.\n\nRegarding your questions, we would address them below:\n\n1. The time it takes to process a single tweet is given for an old CPU only machine to showcase that \neven in circumstances where more complex architectures are not available, the model can operate in \nthose situations with a decent processing speed. We do agree that we should have included the time it takes to run on a  GPU as well. The results of running the system on a GPU (Nvidia Tesla P100) is as follows. \n  \\\\(T_{total} = 400ms\\\\)\n\n    We clearly see that it takes 1/4th of the time that it did on a CPU only machine. Furthermore when the system is deployed on a production level, it would employ multiple instances and auto scaling measures which would increase  the throughput of the system by bounds. The throughput mentioned above is the time it takes to process one tweet by one instance.\n\n\n2. The reason we haven't compared with other papers is that, there doesn't exist exact \nexperiments on the classes or the sub divisions which we divide our data into. We have introduced a newer category while assessing damage as well as proposed how our output could be used to assign priorities. There are papers which deal with the subset of what we are classifying and any comparison will not be an exact one.\n    But it is definitely a great addition to compare our results with some similar experiments performed by others.\n    We cannot compare the overall F1 score of the system with others as the tasks they perform are \nlimited or different in their scope , but we can do somewhat of a task by task comparison.\n\n|      Task      | Model                                                | Precision | Recall | Accuracy  | F1-Score |\n|:--------------:|------------------------------------------------------|-----------|--------|-----------|----------|\n| Task 1         | Multi Modal Logistic  Regression Decision Policy [1] |           |        | 0.80      |          |\n|                | Unimodal Text Pipeline [2]                           | 76        | 76     | 75.9      | 76       |\n|                | **Multi-modal BERT-ResNet (Ours)**                   | **83**    | **77** | **86**    | **79**   |\n| Task 3         | SES-Cross-BERT-DenseNet [3]                          |           |        | 72.65     | 59.76    |\n| (Damage Class) | **Multi Task DeIT-ALBERT (Ours)**                    | **84**    | **88** | **92**    | **86**   |\n| Task 4         | MultiModal CNN-VGG16 [4]                             | 78.5      | 78     | 78.4      | 78.3     |\n|                | SES-Cross-BERT-DenseNet [3]                          |           |        | **91.14** | 68.41    |\n|                | **Inception-BERT (Ours)**                            | **88**    | **84** | 89        | **86**   |\n\n*Comparison with State of the art*\n\nOverall our system with an F1 score of 83% outperforms the models mentioned above and produces much more fine grained classifications offering much more details to the end teams.\n\n3. We used SMOTE only in Task 1 for texts and not for Images as the they are not very effective when dealing with images as often the images formed could be unrealistic and as a result hinder the learning. Also SMOTE works fine with text data as the feature vectors can be  synthesized based on other minority class and help in the learning whereas in images multiple parameters like spatial information, Visual Interpretation could be altered and result in lower quality data. We stuck with class weights in the remaining tasks as we found the results promising when using them as compared to SMOTE as the class weights force the model to pay more attention to minority class during training by assigning higher penalty for wrong predictions on minority class. \n\n4. While the accuracy of the multimodal models hasn't increased by a huge margin, overall the quality of the predictions has, as mentioned in the results. For example in Task 2 the text pipeline gave better results for the Humanitarian class while the image pipeline had greater metrics for the Damage class. Upon combination, we saw a balancing effect wherein the Precision and recall for both the classes reached a similar state and a cancelling of the earlier bias towards a class.\n  Nonetheless, The reason for accuracy not improving much could be attributed to the fact that, the image is unrelated to the text, or the textual information presents a greater confidence to a certain class  than offered by image. Also sometimes the image could be blurred or just blank, or in general poor image quality could lead to lesser inference from the image. But in general , as mentioned earlier, We see improvement on a larger picture when utilizing multiple modalities. \n\nWe will fix all the redundant spaces in the final version of the paper. Regarding Reproducibility, we have specified the parameters of the core model in the paper itself in the experiments and final system discussion headings along with a mention of setting for all the experiments and relevant code in the supplementary materials. The datasets are publicly available, except the structure annotation performed on a subset of the CrisisMMD data, which can be made available on request.\n\n**References**\n\n*[1] A. Gautam, et al., \"Multimodal Analysis of Disaster Tweets,\" in 2019 IEEE Fifth International Conference on Multimedia Big Data (BigMM), Singapore, Singapore, 2019 pp. 94-103. doi: 10.1109/BigMM.2019.00-38*\n\n*[2] S. Madichetty and M. Sridevi, \"Detecting Informative Tweets during Disaster using Deep Neural Networks,\" 2019 11th International Conference on Communication Systems & Networks (COMSNETS), Bengaluru, India, 2019, pp. 709-713, doi: 10.1109/COMSNETS.2019.8711095.*\n\n*[3] Mahdi Abavisani, Liwei Wu, Shengli Hu, Joel Tetreault, Alejandro Jaimes; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 14679-14689*\n\n*[4] Ofli, Ferda et al. “Analysis of Social Media Data using Multimodal Deep Learning for Disaster Response.” International Conference on Information Systems for Crisis Response and Management (2020).*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The whole system is very interesting to deal with crisis events like natural disasters. Although is not responsibility of the author, the elimination of the API of twitter impossibilities the real performance of this systems.", "reasons_to_reject": "The authors mentioned, in line 586, they have a reasonable time to execute the whole system, calculating on average 1.32 seconds per tweet. Honestly, the whole community produces a lot of tweets per second, and when we dealing with natural disasters quick response is crucial, so in my opinion, 1.32 seconds per tweet is a lot of time. So, be careful with this kind of conclusion or insight. \n\nSecond, the authors stated in line 572, they reached an overall f1-score from the whole system of 83%, which I agreed with it seems very impressive, but maybe it would be nice to compare it with other related work, I know that maybe there are no systems similar in the literature, but systems dealing with a subtask of your system yes, so it is important to locate your contribution into the state of the art.", "questions_to_authors": "The authors mentioned, in line 586, they have a reasonable time to execute the whole system, calculating on average 1.32 seconds per tweet. Honestly, the Twitter community produces a lot of tweets per second, and when we dealing with natural disasters quick response is crucial, so in my opinion, 1.32 seconds per tweet is a lot of time. So, be careful with this kind of conclusion or insight. \n\nSecond, the authors stated in line 572, they reached an overall f1-score from the whole system of 83%, which I agreed with it seems very impressive, but maybe it would be nice to compare it with other related work, I know that maybe there are no systems similar in the literature, but systems dealing with a subtask of your system yes, so it is important to locate your contribution into the state of the art.\nSo, I will appreciate some addition to this in the manuscript. \n\nMinor issues:\nFigure 1 is very low quality, it is very difficult to appreciate it. \nWhy used SMOTE in textual tweets and don't in the other kind of task? For instance, in task 4, you used class weights to deal with class imbalance.\nCould you give some ideas about why multimodal is not getting better results (with a large margin) versus text only?", "paper_topic_and_main_contributions": "The paper presents a multimodal system that classifies tweets (text and image) into different categories in a hierarchical way. The whole system is organized in separated tasks, four tasks in total with dependency between them.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "The authors have a lot of missing white spaces and also, extra white spaces.\nFor instance see lines: 17, 135, 352, 569, 453, 419, and more. "}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper deals with approaches based on multimodal datasets in order to propose sequential hierarchical classification. The hierarchical model can give results at different abstraction levels based on dedicated use cases. This paper highlights how the proposed system is effective in classifying multimodal tweets. All reviewers appreciated the soundness and excitement of this paper. This paper has great potential for acceptance for presentation at main conference.

Here is a summary of the major points in the reviews.

Reasons To Accept:
- The motivations to exploit social media and multimodality in disaster assessment, management and analysis of the severity of the damage and humanitarian sub-tasks in real time are convincing.
- The proposed system is very interesting to deal with crisis events like natural disasters. 
- The new solid augmented dataset for the multi modal task is constructed.
- The nice pipeline for putting together existing tools for a common goal which has the four subtasks is proposed.
- Experiments are conducted with several state-of-the-art text and image embeddings.
- The experiment result shows the proposed multimodal-processing approach marginally improve performance.

Reasons To Reject:
- The used tequniques are not really original (but the paper proposes an attractive combination of up-to-date methods well evaluated)
Decision: Accept-Main