A Unified Framework for Synaesthesia Analysis
Authors: Kun Sheng
Zhongqing Wang
Qingqing Zhao
Xiaotong Jiang
Guodong Zhou
Venue: EMNLP 2023 Findings (2023)
Keywords: Synaesthesia Analysis
Generation Model
Lingusitic

Abstract: Synaesthesia refers to the description of perceptions in one sensory modality through concepts from other modalities. It involves not only a linguistic phenomenon, but also a cognitive phenomenon structuring human thought and action, which makes understanding it challenging. As a means of cognition, synaesthesia is rendered by more than sensory modalities, cue and stimulus can also play an important role in expressing and understanding it. In addition, understanding synaesthesia involves many cognitive efforts, such as identifying the semantic relationship between sensory words and modalities. Therefore, we propose a unified framework focusing on annotating all kinds of synaesthetic elements and fully exploring the relationship among them. In particular, we introduce a new annotation scheme, including sensory modalities as well as their cues and stimuli, which facilitate understanding synaesthetic information collectively. We further design a structure generation model to capture the relations among synaesthetic elements and generate them jointly. Through extensive experiments, the importance of proposed dataset can be verified by the statistics and progressive performances. In addition, our proposed model yields state-of-the-art results, demonstrating its effectiveness.

A Unified Framework for Synaesthesia Analysis
Kun Sheng', Zhongqing Wang', Qingqing Zhao², Xiaotong Jiang' and Guodong Zhou'
'Natural Language Processing Lab, Soochow University Institute of Linguistics, Chinese Academy of Social Sciences (ksheng_22 devjiangl (ooulooo com, zhaoqge cass cn (wangzq, gdzhou @suda. edu.
cn cn
Abstract
Sensory werd
(Taoe E> Heeoiing)
Synaesthesia refers to the description of per- ceptions in sensory modality through cepts from other modalities. It involves not only a linguistic phenomenon, but also cog- nitive phenomenon structuring human thought and action, which makes understanding challenging- As a means of cognition, synaes- thesia is rendered by more than sensory modal- ities, cue and stimulus can also play an im- portant role in expressing and understanding it. In addition, understanding synaesthesia in- volves many cognitive efforts, such as identi- fying the semantic relationship between sory words and modalities. Therefore, we pro- pose unified framework focusing on annotat- ing all kinds of synaesthetic elements and fully exploring the relationship among them. In par- ticular, we introduce new annotation scheme, including sensory modalities as well as their cucs and stimuli, which facilitate understand- ing synaesthetic information collectively. We further design structure generation model to capture relations among synaesthetic ements and generate them jointly. Through extensive experiments, the importance of the proposed dataset can be verified by statis- ties and progressive performances. In addi- tion, our proposed model yields state-of-the results, demonstrating its effectiveness.
Stimulss
She soid in sweet tone, "That's what taught him!""
Cluc
structure generation model
ROOT
Original
Synsechetic
Word
Sensory Modality
Cue
Stimalus
Sensory Modality
Sweet
Taste
Tone
Said
Hearing
Figure An example of the proposed unified frame- work for synaesthesia annotation and generation.
2020). For instance, as shown in Figure the gus- tatory adjective "sweet" can be used to describe an auditory perception, in the phrase "said in sweet tone".
There are extensive studies on linguistic synaes- thesia from various perspectives (Shen, 1997; Winter, 2019; Zhao et al., 2022). Nevertheless, synaesthesia has received little attention in nat- ural language processing. As one of the initial study, Jiang et al. (2022) constructed human- annotated Chinese synaesthesia dataset, where sensory words, original and synaesthetic sensory modalities are annotated. They further designed pipeline system identify the sensory word, and detect the original and synaesthetic sensory modalities for the sensory word.
1 Introduction
Synaesthesia, based on the Greek roots 'syn' (to- gether) and "aisthesia" (perception), describes situation in which perceptions in different sen- sory modalities are associated in both percep- tual experiences and verbal expressions (Cytowic, 1993; Popova, 2008; Shen and Aisenman, 2008). Synaesthesia in verbal expressions usage whereby lexical items in one sensory modal- ity are employed to describe perceptions an- other sensory modality (Zhao et al., 2018; Zhao,
As means of cognition and communication, synaesthesia is rendered by more than sensory modalities, cue and stimulus can also contribute to synaesthetic concepuualization. Cue and stimu- lus are the trigger words or expressions that cause the synaesthetic usages. As shown in Figure 1, since the cue "tone" and the stimulus "said" are both auditory words, it easy find that "sweet" transferred from taste to hearing. addition, understanding synacsthesia involves many cogni- tive efforts, such as identifying the semantic re-
language
Corresponding author

---

lationship between sensory words and modalities, well interpreting sensory modalities with the cue and the stimulus, which might be difficult for computers to deal with.
ployed in previous methods. Recently, Jiang et al. (2022) constructed a human-annotated Chinese synaesthesia dataset. They further proposed pipeline system to iden- tify sensory words, and to detect the original and synaesthetic sensory modalities. However, the an- notated samples in their dataset is limited, and cue and stimulus are missing in their annotation scheme. Therefore, we propose a unified frame- work focusing on annotating all kinds of synaes- thetic elements and fully exploring the relation- ship among them. In particular, we propose a new annotation scheme, including sensory modalities as well as their cues and stimuli, which are helpful for understanding synaesthesia information. We further employ structure generation model fully explore the relations between synaesthetic el- ements and to generate them jointly.
In this study, we propose unified framework address the above challenges by annotating all kinds of synaesthetic elements and fully explor- ing the relationship among them. In particular, we firstly extend the previous dataset by collect- ing more samples from social media. We then pro- pose a new annotation framework, including sen- sory modalities as well as their cues and stimuli, which facilitate understanding synaesthesia infor- mation and help to improve the performance of automatic synaesthesia comprehension systems. Furthermore, we propose a structure generation model capture the relations among synaesthetic elements, and generate all elements jointly. Af- terwards, we employ a structure acquisition and composition based framework to improve the gen- eration model by capturing the interdependency among synaesthetic elements.
Data Annotation and Analysis
In this section, we first give a new scheme of synaesthesia. We then show the collection and an- notation process of the dataset. After that, we give some fundamental statistics and analyses.
The statistics show the potential and usefulness the new annotation scheme. In addition, the ex- perimental results demonstrate the effectiveness of the proposed model. We release our dataset named CSyna at
3.1 Scheme of Annotation
In this study, we propose unified annotation scheme, which aims investigate the capabili- ties of automatic systems not only to distinguish between sensory modalities, but also to capture their semantic constituents. Following the task of semantic role labeling (Gildea and Jurafsky, 2002), we aims to answer the question as "Who express What sensory modalities, towards Whom and Why?" (Màrquez al., 2008: Campagnano et al., 2022). We thus take a subset of semantic roles, namely, cue and stimulus, to identify who or what the participants are in an action or event denoted by a sensory modality. The following is the description of the annotation scheme:
2
Related Works
Studies on linguistic synaesthesia from linguistic perspective focus on the directionality patterns and underlying mechanisms for synaesthetic transfers between different modalities. Note that "synaes- thesia" and "synesthesia" are used interchange- ably in the literature. For consistency, we use "synaesthesia" in this paper. For instance, pre- vious studies (Ullmann, 1957; Williams, 1976; Strik Lievers, 2015; Zhao et al., 2019) found that the transfers of linguistic synaesthesia conformed certain patterss, rather than mapping randomly. terms of the mechanisms underlying synaes- thetic transfers, Zhao et al. (2018) and Winter (2019) have suggested that linguistic synaesthe- sia is grounded in multiple mechanisms. In ad- dition, Strik Lievers et al. (2013) and Strik Liev- ers and Huang (2016) focus identifying linguis- tic synaesthetic expressions in natural languages. However, their studies are conducted by semi- automatic methods with lots of manual strate- gies. There are no comprehensive computational models with automatic synaesthesia detection em-
Sensory Word is an adjective that expresses sensory modality in sentence.
Original Sensory Modality is the original domain of sensory word, which is always stationary, and not influenced by the context of the sensory word. The five 'Aristotelian' senses (Strik Lievers, 2015; Winter, 2019; Zhao, 2020), including Touch, Taste, Smell, Vision, and Hearing, are used this study.

---

Synaesthetic Sensory Modality is the actual sensory modality of sensory word, which is influecedd by the sensory word and its con-
#Sentence
24,000
#Sensory Word
236
#Cue
2,397
text.
#Stimulus
706
#Avg. Cue Per Sentence
0.994
Cue is a trigger word or an expression modi- fied by sensory word.
#Avg. Stimulus Per Sentence
0.304
Table A summary statistics of the dataset.
Stimulus is an entity, action or event that causes the synaesthetic sensory modality identified by the cue.
Original
Synaesthetic
Vision
4,901
5,103
As shown in Figure 1, the sensory word "sweet" expresses the synaesthetic sensory modality hear- ing, which is different from its original sensory modality taste. In addition, the synaesthetic sen- sory modality is caused by its cue "tone" and stim- ulus "said". Therefore, the new scheme gives a comprehensive study of synaesthesia, which can be used to explore what and why synaesthesia would happen in sentence. Additionally, it facil- itates understanding synaesthesia information and helps to improve the performance of automatic synaesthesia comprehension systems.
Hearing
38
5,221
Taste
3,042
240
Touch
3,976
1.194
Smell
43
242
Table 2: Statistics of original and synaesthetic sensory modalities.
plcs. We also held regular meetings to discuss an- notation problems and matters that needed atten- tion. The guidelines changed three times when new problems emerged or good improvement methods were found. The kappa score was used to measure inter-annotator agreements (Fleiss, 1971). The agreement on identification of sensory modality was k = 0.78; detection of cue and stim- ulus was k = 0.71, which means they are substan- tially reliable.
3.2 Data Collection and Annotation
With the goal of creating a large-scale synaes- thesia dataset to support research on understand- ing synaesthesia, we extend Jiang et al. (2022)'s dataset by collecting more samples from social media. Each sentence is ensured to contain at least sensory adjective as the candidate sensory sen- tence. The detail statistic of the dataset can be found in Section 3.3.
3.3 Statistic and Analysis
We firstly give the summary statistics of the dataset in Table 1. There are 24,000 sentences with 236 sensory words in the proposed new dataset, which is much larger than the previous dataset (Jiang et al., 2022). In addition, we also annotate 2,397 cues and 706 stimuli in the dataset. On average, each sentence includes 0.994 cue and 0.304 stimulus. It shows that most of sentences stimulus.
Following the new scheme, the annotation of synaesthesia includes the original sensory modal- ity, the synaesthetic sensory modality, the sen- sory word, the cue and the stimulus. In par- ticular, we invited expert annotators to complete this challenging annotation task, which required a relatively deep understanding of synaesthetic units. The annotator team comprised five annota- tors who are postgraduate student researchers ma- joring in computational linguistics with synaesthe- sia or metaphor study backgrounds. The anno- tators formed groups of two, plus one extra per- son. We used cross-validation in annotation pro- cess: the two-member groups annotated, and the fifth person intervened if they disagreed.
contain least
We then analyze the distribution of original and synaesthetic sensory modalities in Table Among the five sensory modalities, vision and touch are the most frequent original sensory modalities. In addition, although synaesthesia rarely occurs in the auditory modality, the visual modality tends to transfer to the auditory modal- ity. Thus, synaesthetic sentences with the auditory modality have the largest number (The examples can found in Figure 2).
Annotations of synaesthesia with multiple in- formation rely on annotators' introspection, which might be subjective. Therefore, we added guide- line course, detailed instructions, and many sam-
Furthermore, we give the transfer probability
between the original and synaesthetic modalities,

---

Original Sensory Modaliry
Sysaestheck Sensory Medallit
Transfer Prubability
Seasory Ward
Cac
Slimolas
Examuple
big
sound æ
shosted [
Secing with face full of and anger, he shouted with lig anund 2t. 88,8 d5 &. 166
Rearing
(36334901)
74.13%
low
stout <2
said
she saud Michael in low sbout. 1e.12-4 12.0.A6f T.t.
Vieion
fighlly
1yee 80
cloed M
closed my ryes tightly bit my lower ltp. A. M (.M. AL
Touch
21.14%
(10366400))
loose
Bloor +8-4
crocked 8
The foox las long becs loose and crackod. 18-K + 2.
stump
ga2e E.A.
sweeping 1P-1
selt that was sweeping the mouncatts with starp gaze. & 24 M.
Viaion
63.54%
SR.
Touch
(2447:3976)
south
We sew 4.
big eyes and pointy mouth.
kof
Vince ++
callad oun +]
Anna strend up colled in voice. 2 am 45 42.
Masring
2--2 1
(1328-2976)
Voice #-
heard Bg
lieand the inciuted and bankh voice of the fiihht atiendant. 6, N. 18 I
18.06
fresh
yellow x
apoeorance A-11
Cologre plone boodhs one fnsh yellow and have very stroamiined appearenct.
Tasiz
83.66% (2551:3042)
++ 19:216 2.16 it, 1::3 q.B.
grorn
green
Figure 2: Transfer probabilities and examples between original and synaesthetic modalities.
ROOT
She said in sweet tone, "That's taught him!" sentence
Tree Generation Model
Original
Sy Syaeshhetic
Word
Suevay
Cuc
Stimulus
Senwry
Hearing
Structure Acquisitin and Composition
Synaesthetic Tree
Linearization:
(ROOT, Original ([EENSOYY WORD) Sweet [ORIGINAL SENSORY MODALITY Taste).
Symsesthetic (CCUE) [STIMULUS] [SYNAESTHETIC MODALITY Heuring
Figure 3: Overview of the synaesthetic structure generation model.
along with examples in Figure 2. Since there too many combinations, just show the most frequent original and synaesthetic pairs in the fig- ure. Based on the synaesthesia transfer probabil- ity, we find that: tactile adjectives are the most likely to be used for vision, with the transfer prob- ability of 61.5%. The association between vision and hearing is similar to that between touch and vision. Specifically speaking. visual adjectives are the most likely be associated with hearing. Meanwhile, also find that the synaesthetic sory modalities are deeply influenced by the cue and stimulus. For example, although "harsh" is tactile adjective, "voice" and "hear" trigger to express hearing. In addition, "appearance" makes the sensory modality of "fresh" transferring from taste to vision.
ments from sentence. As shown in Figure 3, we firstly convert all the synaesthetic elements into tree structure. We then employ a tree generation model to explore the relations among synaesthetic elements and generate them jointly. Afterwards, we utilize a structure acquisition and composition based framework to improve the structure gen- eration model by capturing the interdependency among synaesthetic elements. In this section, we give the description of synaesthetic tree construc- tion and the structure tree generation model, and then discuss the structure acquisition and compo- sition framework in the next section.
4.1 Synaesthetic Tree Construction
As shown in the right side of Figure 3, the synaes- thetic tree models sentence using rooted di- rected acyclic graph, highlighting its main ele- ments (e.g. sensory word, cue, stimulus) and re- lations. Given a sentence, we convert synaesthetic elements into the synaesthetic tree as below:
4 Synaesthetic Structure Generation
Given a sentence, we employ a structure gener- ation model to generate all the synaesthetic ele-

---

We create the original and synaesthetic node to represent the original and synaesthetic sen- sory information respectively, and connect them with a virtual root node;
vectors. At the i-th step of generation, the self- attention decoder predicts the i-th token yi in the linearized form, and decoder state hg as:
34, h - Deecoder([H;)))) ..y h2-1l. v6-1) (2)
The sensory word and the original sensory modality are linked to the original node as leaves.
where each layer of Decoder a transformer block that contains self-attention with decoder state hg and cross-attention with encoder state H.
The synaesthetic sensory modality, the cue and the stimulus are linked the synaesthetic node leaves.
The generated output structured sequence starts from the start token "(bos)" and ends with the end token "(cos)". The conditional probability of the whole output sequence p(y(x) is progres- sively combined by the probability of each step p(yiluer
Therefore, the tree structure is very important for understanding synaesthesia information and learning the correlations among synaesthetic el- ements. For instance, the connections between the sensory word and the original sensory modal- ity can be used to identify the sensory word. In addition, the sub-tree of the synaesthetic sensory modality is helpful for detecting the synaesthetic sensory modality based on the cue and the stimu- lus.
lvl p(ulx) II p(ulyee,
(3)
where y<i = y1.--Wi-., and p(v: M(silves, x) are the probabilities over target vocabulary V normalized by softmax.
Since it is much easier to generate sequence than tree (Vinyals et al., 2015; Lu et al., 2021), we linearize the synaesthetic tree to the target se- quence. As shown in the bottom of Figure 3, we linearize tree into token sequence via depth- first traversal, where "(" and """ are structure in- dicators used represent the structure of linear expressions.
Since tokens lincarized representations are also natural language words, we adopt the pre- trained language model T5 (Raffel et al., 2020) as our transformer-base encoder-decoder archi- tecture. In this way, the general text generation knowledge can be directly reused.
4.2 Synaesthetic Tree Generation
4.3 Objective Functions and Training
We then employ text generation model to gener- ate the linearized synaesthetic tree from the given sentence.
In this subsection, we show the objective functions and the training process the proposed model. The goal is to maximize the output linearized tree Xy probability given the sentence X. There- fore, we optimize the negative log-likelihood loss function:
In this study, we employ a sequence-to- sequence model to generate the synaesthetic tree via transformer-basel encoder-decoder architec- ture (Vaswani al., 2017). Given the token quence 31, ...,.... xx1 2| input, the sequence-to- sequence model outputs the linearized representa- tion y = yı..... M/yl To this end, the sequence-to- sequence model first computes the hidden vector representation H = hı,..., h of the input sen- tence via multi-layer transformer encoder:
C
log logp(XrlXx:0) (X.XT)
Σ
(4)
|
where 0 is the model parameters, and Xx) is (sentence, tree) pair in training set then
logp(Xr(X;00)
H Encoder( Fi())
(1)
n
(5)
where each layer of Encoder is transformer block with the multi-head attention mechanism. After the input token sequence is encoded, the decoder predicts the output tree structure token- by-token with the sequential input tokens' hidden
logp(ar) pay, 2 ...... + x:0)
calculated by where p(a | x ...II .x:) the decoder.

---

Struture Acquistion
(ROOT ISENSORY WORD)>SSeesee
[ROOT:
DriginalSENNSORY WORD),
[ORIGINAL SENSORY I) -Settence
(ROOT:
OriginalISENSORY WORD)
[ORIGINAL SENSORY NOODALITTIMCUUE)-SSUnNN
Structure Generation Model
(ROOT,
OriSY WORD),
(ORIGINAL, SENSORY MODALITY].
Synasthric([CUEER
Struture Compositon
(ROOL,
Originalt ISENSORY WORD)
(ORIGINAL SENSORY MODALITY])
(SYNAFSTHETICC SENSORY MODALITY]]-
Synaesthetic Trees
Original
Synaesthetic
Figure 4: example of structure acquisition and composition.
5 Structure Acquisition and Composition
ements. Then we construct task prompt for each subtask based on the elements it contains. For example, The special token corresponding to WENSORY_WORRE is "[ SENSORY WORD 1". The input for each subtask simply adds a task prompt to the input for the original task.
Since the alignment between sentences and synaesthetic elements, and the interdependency among synaesthetic elements are very important for synaesthesia analysis, we then propose novel strategy to improve the structure generation model by addressing the above issues. As shown in Figure 4, we firstly decompose the original task into several subtasks. Each subtask corresponds mapping the natural language sentence to one more synaesthetic elements. Afterwards, feature prompt-based learning strategy to sep- arately acquire the structural knowledge of sub- tasks and employ the learned knowledge to tackle the main task, i.c., generating all the synaesthetic elements. In the structure acquisition stage, we train the model with all the subtasks in multi- task learning manner; in the structure composition stage, we fine-tune the model with the main task combine the acquired structural knowledge of subtasks and learn the dependency between them.
5.2 Structure Composition
Training the model with multiple subtasks can- not capture the interdependency between them. In this stage, fine-tune the model with the main task, i.e., generating all the synaesthetic elements to capture such information. As shown in Figure we combine the prompts subtasks to construct the prompt of the main task to guide the model to composite the knowledge of subtasks.
Experiments
In this section, we conduct various experiments to investigate the effectiveness of the proposed method the synaesthetic elements generation task. In addition, give several analyses and dis- cussions show the importance of the proposed new dataset.
5.1 Structure Acquisition
As shown Figure 4, we decompose the synaes- thetic structure generation task into four subtasks. Basically, a subtask aims to translate the sentence to one or more synaesthetic elements. For exam- plc, the SENSORY WORD subtask aims to generate the sensory word given the sentence. We then train the sequence-to-scquence model with all subtasks using multi-task learning. We assign each subtask a special token, which is also used to denote its corresponding synaesthetic el-
6.1 Setting
We evaluate our proposed structure generation model on the new Chinese synaesthesia dataset. There are totally 24.000 sentences in the proposed new dataset. We split the dataset into the train- ing set (80%), the validation (10%) and the test set (10%). should be noted that these sets are separated by sensory words, which means that the

---

Method
Sensory Word
Original 74.2
Synaesthetic
Cue
Stimulus
Overall
BERT+CRF
74.6
70.3
77.4
83.9
53.0
Radical
75.6
75.1
71.4
77.7
84.2
54.3
BART
72.5
68.4
63.8
73.5
79.3
42.
T5
72.8
69.4
65.2
71.3
80.1
40.7
Human
77.1
75.0
73.6
79.2
80.4
49.1
ChatGPT
41.8
22.3
27.0
35.0
49.5
15.3
Ours
78.3
78.3
73.9
79.7
86.3
61.9
Table Comparison with baselines.
sensory words among different sets are totally dif- ferent.
boundary and to jointly classify the original and synaesthetic sensory modalities.
We employ T5' and fine-tune its parameters for generation models. We tune the parameters of our models by grid searching on the validation dataset. We select the best models by early stop- ping using the Accuracy results on the validation dataset. The model parameters are optimized by Adam (Kingma and Ba, 2015) with learning rate of le-4. The batch size is 16. Our experiments are carried out with Nvidia RTX 3090 GPU. The experimental results obtained by averaging ten runs with the random initialization.
BART employs the pre-trained language model BART (Lewis et al., 2020) to generate the synaesthetic elements individually?.
T5 utilizes T5 (Raffel et al., 2020) to generate the synaesthetic elements individually, which be considered as the baseline method of the proposed model.
Human engaged two volunteers to identify and label the synaesthetic elements, and the results are considered as representative hu- man performance.
We use Fl-score as the evaluation metric for the sensory word, cue, and stimulus extraction, and weighted Fl-score (Manning and Schüitze, 1999) as the evaluation metric for the sensory modality detection. In addition, a synaesthetic quintet (sen- sory word, original sensory modality, synaesthetic sensory modality, cue, stimulus) is viewed cor- rect if and only if the five elements, as well as their combinations, are exactly the same as those gold quintet. On this basis, we calculate FI- score (Cai et al., 2021) as the overall evaluation metric for the synaesthesia analysis.
ChatGPT is sibling model Instruct- GPT (Ouyang et al., 2022), which trained to follow an instruction vides a detailed response. We ask to gener- ate the synaesthetic elements from the input sentences
prompt and pro-
From the results, we find that the basic genera- tion model (i.e., BART, T5) cannot give an accept- able result, the performance of which is even lower than BERT+CRF. may be due that the basic generation model generates each element one by one, ignoring the correlations among them. In ad- dition, the performance of ChatGPT is much lower than other models. One of the possible reasons is that we do not fine-tune it on the training data, and we only give some prompts to let it understand the synaesthesia analysis task. Based analyzing its outputs, find that it is hard for ChatGPT pre- dict the original and synaesthetic sensory modali- ties, and it also cannot capture the relations among all these synaesthetic elements.
6.2
Main Results
We firstly compare the proposed structure gener- ation model with various strong basclines on Ta- ble 3, where,
BERT+CRE adopts BERT (Devlin al., 2019) as the textual encoder, followed by a CRF output layer (Radford and Narasimhan, 2018).
Radical (Jiang al., 2022) is pipeline sys- tem, which employs radical-based neural network model identify the sensory word's
bart- base facebook/ /
hittpa://agginngaace.u co/
"httos://huggingge" co/t5-baa
Phttps: //tp///oppea.a//e wateenaa..cmccheeeee com/chat.pt.

---

Method
-l-score
Tuples
Ours 61.9
22)
71.3
(20, o)
76.8
75.7
SAC 60.7
(20,
76.9
71.2
Tree 60.5
[0r, 0,8) 8)
77.2
76.1
71.7
-ACC -Tree
59.8
[2), 80,8,00
77.5
12.7
77.8
(wr, 8,4)
76.9
72.1
83,8
[00, c,
78.3
73.1
77.7
84.8
Table 4: Impacts of different factors in the proposed structure generation model with overall Fl-score surements.
Ours
78.3
78.3
73.9
79.7
86.3
Table 5: Effects of synaesthetic elements.
In contrast, our proposed model outperforms all the previous studies significantly (p < 0.05) in all settings. Compared to native speakers, our pro- posed model surpasses human performance, mak- ing easier generate all five synaesthetic ele- ments correctly. This indicates that the struc- ture much more helpful for capturing the cor- relations among synaesthetic elements. Further- more, the results also indicate the effectiveness of the structure acquisition and composition strat- egy, which is used to learn alignment between sen- tences and synaesthetic elements, and the interde- pendency among synaesthetic elements. The de- tails of the case study shown in Appendix A.
the cue and the stimulus respectively. In addi- tion, the rows in the table mean different com- binations of synaesthetic elements. For example, (w,, 8) means that we used the proposed structure generation model to generate the sensory word and the synaesthetic sensory modality jointly.
From the table, we can find that: 1) The sensory modalities are deeply correlated with the sensory words. If we jointly generate the sensory word with original (w, 0) or synaesthetic sensory modal- ities (w, s), the performance of the sensory word extraction is improved more than 5%. 2) The orig- inal and synaesthetic sensory modalities are cor- related, and the joint prediction (w, 0, 8) is better than predicting them individually. 3) Cue and stimulus are both helpful for extracting the sen- sory word and predicting the synaesthetic sensory modality. we integrate them into the generation model, the performance of the sensory word ex- traction and the synaesthetic sensory modality pre- diction are higher than before. 4) When we gen- erating all the elements jointly (ours), the perfor- mance of the proposed model can achieve the best performance.
6.3 Impact of Different Factors
As shown Table 4, we employ ablation experi- ments to analyze the impact of different factors in the proposed structure generation model.
If we remove the structure acquisition and com- position strategy (-SAC), the performance drops 60.7%. It indicates that this strategy is very im- portant for generating a valid tree and capturing the interdependency among synaesthetic elements. If we remove the tree structure, and only gener- ate the flat sequence, the performance drops to 60.5%. It shows that the tree structure is beneficial capture the correlations between synaesthetic elements. Furthermore, effective than the sim- posed model is much more ple text generation model. If we remove both the structure acquisition and composition strategy and the tree structure, the proposed model degrades to a T5-based sequence-to-seqeene model, and the performance drops to 59.8%.
summary, the above observations show that all the synaesthetic elements are correlated, and the structure generation model is helpful for gener- ating them jointly. Furthermore, the results among subtasks (i.e., ().o). (w, s)c) and the main task (ours) also indicate the importance of the struc- ture acquisition and composition strategy, which is used learn from all the subtasks to the main task.
also find that the pro-
Conclusion
Synacsthesia refers to the description of percep- tions in one sensory modality through concepts involves not only
6.4 Effects of Synaesthetic Elements
We then analyze the effect of synaesthetic ele- ments in Table 5, where w denotes the sensory word, o and s are the original and synaesthetic sensory modalities respectively, and e and are
from other modalities. linguistic phenomenon, but also a cognitive phe- nomenon structuring human thought and action, which makes understanding challenging and re-

---

warding. In this study, we give comprehensive study on the synaesthesia analysis from compu- tational perspective. In particular, we firstly in- troduce a new annotation framework, including annotating the sensory modality as well as their cues and stimuli, which facilitates understanding the synaesthesia information. We further design a structure generation model to fully explore the relations among synaesthetic elements and gen- erate them jointly. The statistics show the poten- tial and usefulness of the proposed new dataset. In addition, the experimental results demonstrate the effectiveness the proposed model.
of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171 1-61-4186, Minneapolis, Minnesota. Associ- ation for Computational Linguistics.
Joseph I., Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bul- letin, 76:378-382.
Daniel Gildea and Daniel Jurafsky. 2002. Automatic labeling of semantic roles. Comput. Linguistics, 28(3):245-285 288.
Xiaotong Jiang, Qingging Zhao, Yunfei Long, and Zhongqing Wang. 2022. Chinese synesthesia detec- tion: New dataset and models. Findings of the As- sociation for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3877- 3887.
Limitation
Our work focuses employing unified frame- work for the synaesthesia analysis. However, some more general topics should be addressed as future work, such as the metaphor and sen- sory modality analysis. Another limitation is the computational complexity of the proposed model, whose training time is slower than the classification-baxt based models.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA. USA, May 7-9, 2015, Conference Track Proceedings.
Mike Lewis, Yinhan Liu, Naman Goyal, Mar- jan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: denoising sequenne--o--sqquence pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 7871-7880
A Acknowledgements
We would like thank the anonymous reviewers for their insightful and valuable comments. Also, we gratefully acknowledge funding from the Na- tional Natural Science Foundation of China (No. 62076175, No.61976146), and Jiangsu Innovation Doctor Plan.
Yaojie Lu, Hongyu Lin, Jin Xu, Xianpei Han, Jialong Tang, Annan Li, Le Sun, Meng Liao, and Shaoyi Chen. 2021. Text2event: Controllable sequence-to- structure generation for end-to-end event extraction. ACL/JCCNPP 2021, pages 2795-280K
References
Hongjie Cai, Rui Xia, and Jianfei Yu. 2021. supecc-categorryooinioon---eeeee
Christopher D. Manning and Hinrich Schützee 1999. Foundations of Statistical Natural Language Pro- cessing. MIT Press, Cambridge, MA, USA.
quadruple ex- traction with implicit aspects and opinions. In ACL/ICCNP 2021, pages 340-350.
Lluís Mårquez, Xavier Carreras, Kenneth C. Litkowski, and Suzanne Stevenson. 2008. Se- mantie role labeling: An introduction to the special issue. Comput. Linguistics, 1452).145-159.
Cesare Campagnano, Simone Conia, and Roberto Nav- igli. 2022. SRL4E semantic role labeling for emo- tions: A unified evaluation framework. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 4586-4601.
Long Ouyang. Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman. Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instruc- tions with human feedback. CoRR, abs/2203.02155.
Richard E Cytowic. 1993, The man who tasted shapes: bizarre medical mystery offers revolutionary in- sights into emotions. Reasoning, and Conscious- ness.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference
Yanna Popova. 2008. Image schemas and verbal synaesthesia. In From Perception to Meaning, pages 395-420. De Gruyter Mouton.

---

Alec Radford and Karthik Narasimhan. 2018. Im- proving language understanding by generative pre- training.
Qingqing Zhao. Chu-Ren Huang. and Kathleen Ahrens. 2019. Directionality of linguistic synes- thesia in mandarin: A corpus-based study. Lingua, 232:102744.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. Journal of Machine Learning Research, 21(140):1-67.
Qingqing Zhao, Chu-Ren Huang, and Yunfei Long. 2018. Synaesthesia in Chinese: A corpus-based study gustatory adjectives in mandarin. Linguis- rics, 5665):1167-11944
A
Case Study
Yeshayahu Shen. 1997. Cognitive constraints on poetic figures. Cognitive Linguistics, 8:33-71.
Yeshayahu Shen and Ravid Aisenman. 2008. Heard melodies are sweet, but those unheard are sweeter': Synaesthetic metaphors and cognition. Language and Literature, 17(2):007-21.
Examplel: Together, every person responded with warm sound of hitting hands.
* x
F7 22
VA 61 #5 #8.
Francesca Strik Lievers. 2015. Synaesthesia:
every
Vision
sound
corpus-based study of cross-modal directionality. Functions of Language, 22.
Baseline
*
X
Hearing
sound of hitting hands T #
Francesca Strik Lievers and Chu-Ren Huang. 2016 lexicon of perception for the identification of synaesthetic metaphors in corpora. In Proceedings of the Tenth International Conference on Language Resources and Evaluation LREC 2016, Portorož, Slovenia, May 23-28, 2016. European Language Re- sources Association (ELRA).
warmh Aiss 3!
Touch
Ours
Hearing
Example2: Huabui suddenly became angry, his face turned red, and he exclaimed with big sound. * Bs St A & 8t. (K. T ak, A # Mg.
lumedd
Touch
sound #
exclaimed iit
Francesca Strik Lievers, Ge Xu, and Hongzhi Xu. 2013. "a methodology for the extraction of lexical- ized synaesthesia from corpora". In 19th inter- national congress of linguists.
Baseline
Alt
Hearing
x
big *
Vision
sound #
exclaimed
Ours
Hearing
v
Stephen Ullmann. 1957. The principles semantics. Oxford: Busil Blackwell.
Example3: Longtei said with bitter expression the face.
Ht. +
iñ.
said
bitter $
Hearing
face
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. In NIPS 2017, pages 5998-6008.
Baseline
Taste
bitter
Vision
face
Ours
Taste
Oriol Vinyals, Lukasz Kaiser, Terry Koo, Slav Petrov, Tlya Sutskever, and Geoffrey E. Hinton. 2015.
Figure 5: The examples of case study, True predictions marked with v while false predictions are marked with x.
Grammar a foreign language. In NIPS 2015, pages 2773-2781.
Joseph Williams. 1976. Synaesthetic adjectives: A possible law of sematic change. Language, 52(2):461-477)
We provide case studies in Figure 5, wherein we select three examples demonstrate the impact of the proposed model compared with baselines.
Bodo Winter. 2019. Sensory Linguistics: Language, perception and metaphor. Converging Evidence Language and Communication Research. John Ben- jamins.
The first example and the second example are both about the effect of capturing the interdepen- dency between sensory words and cues. With the help of tree structure and structure acquisition and composition, the proposed model can generate ac- curate sensory words and cues. Sensory word modifies cue, and the results of baseline models do not adhere this rule.
Qingging Zhao. 2020. Embodied Conceptualization or Neural Realization: A Corpus-Driven Study of Man- darin Synaesthetic Adjectives. Frontiers Chinese Linguistics. Springer Singapore.
Qingqing Zhao, Kathleen Ahrens, and Chu-Ren Huang. 2022. Linguistic synesthesia is metaphor- ical: a cexieal-conceptual account. Cognitive Lin- guistics, 3353)5553-583.
The third example pertains to the connection be- tween stimulus and synaesthetic sensory modal- ity. Although the other three synaesthetic ele-

---

ments are accurate, baseline models fail to com- prehend that the stimulus "said" does not refer to the synaesthetic sensory modality indicated by the cue "face".


Reviews:
[{"review_id": "cIhujWIuDa", "reviewer": "Reviewer_UVPL", "summary": "", "questions": "- Can you elaborate on the steps taken to ensure consistency and minimize subjectivity in annotating multiple synaesthetic elements, given the complexity of the annotation framework?\n- The structure generation model appears to offer improved accuracy, but could you provide more insight into how the model's decisions can be interpreted or visualized to gain a better understanding of its inner workings?\n- In the context of cognitive science and psychology, how might the insights gained from your study contribute to our understanding of the cognitive processes underlying synaesthesia and cross-modal perception?\n\nPlease also see the weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nWe really appreciate your valuable reviews, we will respond to your review one by one as follows: \n\n> How were inter-annotator disagreements resolved, and what measures were taken to ensure consistency?\n\n**Response:** To resolve the inter-annotator disagreements and ensure consistency, we took the below annotation steps:\n1) We first asked a linguistic expert to annotate a set of sentences as exemples.\n2) We then used these examples as a guide to train the annotators.\n3) In the process of trial annotation, we adjusted the guidelines in time when new problems emerged with the help of the linguistic expert. \n4) To resolve the inter-annotator disagreement, we use cross-validation methods for annotation correction.\n5) We use kappa score to measure agreements between annotators.\n6) Additionally, regular meetings were held to further ensure annotation consistency.\n\n> How transferable is the proposed framework to synaesthesia in languages with different syntactic and semantic structures?\n\n**Response:**  In fact, our proposed framework can be transferred in other languages directly, since the synaesthesia phenomenon has common cross-lingual features in expressions [1] [2]. That is, synaesthesia phenomenon in nearly all languages is likely to have the roles in our dataset: stimulus, cue, sensory word and sensory modality. \n\nFor instance, if we have an English synesthesia dataset, we can utilize it to train a pretrained English generative model within our proposed framework. This model will leverage its internal understanding of English syntax and semantics.\n\n[1] Zhao, Q. 2020. Embodied Conceptualization or Neural Realization: A Corpus-Driven Study of Mandarin Synaesthetic Adjectives. Springer, Singapore.\n\n[2] Strik Lievers, F. 2015. Synaesthesia: A corpus-based study of cross-modal directionality. Functions of language, 22(1):69–95.\n\n> How does the proposed model's performance compare to human performance in terms of identifying and generating synaesthetic elements?\n\n**Response:**   First, it should be noted that the task is challenging for not only computers but also human without expert knowledge since there are some difficult cases in synaesthesia phenomenon. And that’s why we want to use a computational linguistic method to do the task. That’s also the reason why our annotators need the guidance from a linguistic expert and regular meetings, even they are all Chinese native speakers. So, from the annotators’ experiences, we believe our methods can reach a comparable performance compared to a native speaker without any expert knowledge training. \n\nSecond, it’s still a very good advice to compare the performances from human and the proposed model, we will add the human performance results in the camera-ready version. \n\n> The structure generation model appears to offer improved accuracy, but could you provide more insight into how the model's decisions can be interpreted or visualized to gain a better understanding of its inner workings?\n\n**Response:**  As a means of cognition and communication, synaesthesia is rendered by more than sensory modalities, cue and stimulus can also contribute to synaesthetic conceptualization. Previous methods fail to take all these roles into consideration, we use a tree structure to model the relationship among all roles in a more structural way. Moreover, we use a generative framework to capture the relations among synaesthetic\nelements, and generate all elements jointly.\n\nSpecifically, since the alignment between sentences and synaesthetic elements, and the interdependency among synaesthetic elements are very important for synaesthesia analysis, we first design a structure acquisition stage to learn the alignment, then we design a structure composition stage to model the interdependency among synaesthetic elements. \n\n> In the context of cognitive science and psychology, how might the insights gained from your study contribute to our understanding of the cognitive processes underlying synaesthesia and cross-modal perception?\n\n**Response:**  The results of this study provide new evidence for lexical conceptual mappings of linguistic synesthesia proposed by Zhao et al. (2022). That is , linguistic synesthesia is also a type of conceptual metaphor, where lexicalized concepts of sensory properties are involved.\n\nZhao, Q., Ahrens, K., and Huang, C.-R. 2022. Linguistic synesthesia is metaphorical: a lexical-conceptual account. Cognitive Linguistics, 33(3):553–583.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- They introduces a new annotation framework that goes beyond previous studies by including sensory modalities, cues, and stimuli. This comprehensive approach provides a more holistic understanding of synaesthesia.\n- The proposed structure generation model captures the relationships between synaesthetic elements and generates them jointly. This approach considers the interdependencies that exist between various elements, enhancing the accuracy of synaesthesia comprehension.\n- The authors extends an existing dataset by collecting more samples from social media, resulting in a larger and more diverse dataset for training and evaluation.\n- The paper presents detailed experimental results that clearly demonstrate the superiority of the proposed model over various baseline methods. The evaluation metrics provide a comprehensive view of model performance.", "reasons_to_reject": "- While the comprehensive annotation framework is a strength, it also introduces complexity and potential subjectivity in annotating multiple synaesthetic elements. How were inter-annotator disagreements resolved, and what measures were taken to ensure consistency?\n- The study focuses on Chinese synaesthesia, and the proposed methods might need adaptation to work effectively with other languages. How transferable is the proposed framework to synaesthesia in languages with different syntactic and semantic structures?\n- The authors acknowledges that understanding synaesthesia involves cognitive efforts, which could be challenging for computers. How does the proposed model's performance compare to human performance in terms of identifying and generating synaesthetic elements?", "questions_to_authors": "- Can you elaborate on the steps taken to ensure consistency and minimize subjectivity in annotating multiple synaesthetic elements, given the complexity of the annotation framework?\n- The structure generation model appears to offer improved accuracy, but could you provide more insight into how the model's decisions can be interpreted or visualized to gain a better understanding of its inner workings?\n- In the context of cognitive science and psychology, how might the insights gained from your study contribute to our understanding of the cognitive processes underlying synaesthesia and cross-modal perception?\n\nPlease also see the weaknesses.", "paper_topic_and_main_contributions": "The authors presents an annotation framework and model for understanding and generating grapheme-color synaesthetic experiences. The framework identifies synaesthetic connections and incorporates diverse sensory cues, enriching comprehension. The model effectively generates synaesthetic elements by considering interdependencies between different components. Through social media data, the model demonstrates superior performance compared to baseline methods. This work contributes significantly to synaesthesia research by offering a nuanced annotation approach and a novel generation model for complex sensory experiences.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "NA", "Justification_for_Ethical_Concerns": "NA"}, "extra_text": {}}, {"review_id": "BxOEyWI3Ov", "reviewer": "Reviewer_q3wD", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nWe really appreciate your valuable reviews, we will respond to your review one by one as follows: \n\n> Are the authors planning to make this dataset publicly available? Please clarify. \n\n**Response:** Yes, we will release the dataset. By extending the size of previous dataset and adding the annotation of cue and stimulus, the dataset we have proposed has been refined and can now be open-sourced for collaborative research on synaesthetic phenomena.\n\n\n> I suggest adding more examples from the dataset (to appendix) to better understand the quality of the annotations. I am happy to revise my scores based on the author 's response in the rebuttal on the dataset release.\n\n**Response:**  Thanks for your valuable suggestion, we will add more examples from the dataset to appendix in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-motivated and well-organized in terms of style and structure. The proposed annotation framework in section 3.1 is interesting and will be useful in fostering more research in comprehending synaesthesia information. The paper performs a detailed analysis in section 6.3 and 6.4 on the impact and effect of synaesthetic elements. Overall, the work is well-written and solid in terms of novelty, new benchmark and experiment analysis.", "reasons_to_reject": "I do not find any major weaknesses with this work. Readability of section 4.3 could be improved with more intuitive explanations. Are the authors planning to make this dataset publicly available? Please clarify. I suggest adding more examples from the dataset (to appendix) to better understand the quality of the annotations. I am happy to revise my scores based on the author 's response in the rebuttal on the dataset release.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper addresses the problem of Synaesthesia which refers to the description of perceptions in one sensory modality through concepts from other modalities. The paper proposes a unified framework for annotation of synaesthetic elements. Specifically, the paper introduces a new annotation scheme that aids in understanding synaesthetic information. In addition, the paper proposes a new model that firstly converts all the synaesthetic elements into a tree structure and then employs a tree generation model to explore the relations among synaesthetic elements and generate them jointly. Experimental results and analysis demonstrate the utility of the proposed benchmark as well as the model.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "POHjhH7ZcF", "reviewer": "Reviewer_oMzF", "summary": "", "questions": "1. Did previous synesthesia detection works only use bert-like classification models? Are there any works that employ generative models?\n\n2. How can synesthesia, cue, and other outputs be placed in the baseline models? Why is there such a performance difference between the proposed method based on T5 and baselines BART, T5?\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nWe really appreciate your valuable reviews, we will respond to your review one by one as follows: \n\n> The method is solid, but generating synesthesia for sentences via T5 seems less innovative. Although I read the related work [1] mentioned in this article, only using bert for synesthesia classification, not a generative model.\n\n**Response:** Our main contribution in the model architecture is the idea of using a unified generative framework, which significantly surpasses all existed synaesthesia related work (Jiang et al., 2020). We only use T5 as a backbone, which can be easily replaced with other trending generative methods directly without big changes. \n\n> The main contribution of this work may be the constructed corpus. It is noted that the work does not mention that data will be open-sourced, and the related work mentioned in this work does not open source data too, which leads to the incompatibility of datasets and the inability to compare methods. This seems to be detrimental to research in the community.\n\n**Response:** For fair comparison, we use the same constructed corpus on all experiments. And this constructed dataset will be open-sourced. By extending the size of previous dataset and adding the annotation of cue and stimulus, the dataset we have proposed has been refined and will be open-sourced for collaborative research on synaesthetic phenomena.\n\n> The author doesn't mention how to place synesthesia, cue and other outputs in the baseline. Note that baselines include generative models such as BART and T5, and the proposed method also uses T5. Since all methods require flattening the output data, why is there such a performance difference between the proposed method based on T5 and baselines BART, T5?\n\n**Response:** Different from the flat sequence generated in Section 6.3 ablation experiment, the baseline BART and T5 generate each synaesthesia element one by one. The five individual synaesthetic element generation tasks do not capture the correlations among them, which is why the performance of basic generation model is even lower than that of BERT+CRF. In order to explore the advantages of co-generating synaesthesia elements, we choose the model of  generating synaesthetic elements individually as basic generation model.\n\n> Did previous synesthesia detection works only use bert-like classification models? Are there any works that employ generative models?\n\n**Response:** Yes, previous synesthesia detection works only use bert-like classification models. We are the first to employ generative models for synaesthesia analysis, so that we can better capture the relationship between synesthesia elements and generate them jointly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written and easy to understand.\n\n2. For the synesthesia detection task, it is a reliable idea to use the generative model T5 to generate Synesthesia, cue, stimulus, etc.\n\n3. Compared with previous work, this work increases the detection of cues and stimulus.", "reasons_to_reject": "1. The method is solid, but generating synesthesia for sentences via T5 seems less innovative. Although I read the related work [1] mentioned in this article, only using bert for synesthesia classification, not a generative model.\n\n2. The main contribution of this work may be the constructed corpus. It is noted that the work does not mention that data will be open-sourced, and the related work mentioned in this work does not open source data too, which leads to the incompatibility of datasets and the inability to compare methods. This seems to be detrimental to research in the community.\n\n3. The author doesn't mention how to place synesthesia, cue and other outputs in the baseline. Note that baselines include generative models such as BART and T5, and the proposed method also uses T5. Since all methods require flattening the output data, why is there such a performance difference between the proposed method based on T5 and baselines BART, T5?\n\n[1] Jiang et al., Chinese Synesthesia Detection: New Dataset and Models", "questions_to_authors": "1. Did previous synesthesia detection works only use bert-like classification models? Are there any works that employ generative models?\n\n2. How can synesthesia, cue, and other outputs be placed in the baseline models? Why is there such a performance difference between the proposed method based on T5 and baselines BART, T5?\n", "paper_topic_and_main_contributions": "This work aims to generate synaesthesia annotations for sentences. First, it builds a synaesthesia corpus. Then, it converts the synaesthesia data into a tree structure and linearizes the tree. Finally, the sentence-to-synaesthesia tree generation is learned by the T5 pre-trained model. In particular, the author proposes to transform the task of generating whole synaesthesia data into multiple subtasks to learn different parts of data to improve accuracy.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper under consideration presents an innovative approach to understanding and generating grapheme-color synesthetic experiences. It introduces a comprehensive annotation framework and a novel model for this purpose. Overall, the work is well-structured, motivated, and addresses an important topic in synesthesia research. It makes significant contributions to the field, but there are also some potential concerns that need to be addressed.

Pros:

- The paper introduces a novel annotation framework that includes sensory modalities, cues, and stimuli, providing a holistic understanding of synesthesia.
- The proposed model effectively captures the interdependencies between synesthetic elements, enhancing comprehension accuracy.
- The expansion of the dataset through social media data collection contributes to a more diverse and robust dataset.
- The experimental results clearly demonstrate the superiority of the proposed model over baseline methods, providing a comprehensive view of its performance.

Cons:

- The comprehensive annotation framework may introduce complexity and subjectivity. The paper lacks information on how inter-annotator disagreements were resolved and how consistency was ensured.
- The study focuses on Chinese synesthesia, and the generalizability of the proposed framework to other languages with different structures is unclear. Moreover, the title does not specify that the study focusses on Chinese only
- The paper acknowledges that understanding synesthesia involves cognitive efforts. A comparison of the proposed model's performance to human performance in identifying and generating synesthetic elements is missing.
- Section 4.3 could be improved for better readability, and the authors should clarify their plans for making the dataset publicly available to foster research.
- While the paper is well-structured, the use of T5 for synesthesia generation might be considered less innovative in comparison to other generative models.

In summary, the paper makes notable contributions to synesthesia research with its comprehensive framework and model, but it should address concerns related to subjectivity in annotation, cross-lingual adaptability, and comparisons to human performance. Additionally, improvements in readability and dataset availability are recommended.

In the rebuttals, the authors were able to provide access to the data and explain many of the issues raised by the reviewers, yet the paper would need some substantial edits.
Decision: Accept-Findings