RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Methodology
In this section, we first describe the problem statement of conversational recommendation systems (CRS). Then we present the proposed P arameter- E fficient C onversational R ecommender S ystem ( PECRS ) method in detail. The overall architecture of PECRS is shown in Figure 2 .
Problem Formulation
Let \bm{\mathcal{I}}=\{I_{1},I_{2},\ldots,I_{N_{\text{item}}}\} represent the item database, which contains N_{\text{item}} unique items, and \bm{\mathcal{D}}=\{D_{1},D_{2},\ldots,D_{N_{\text{dial}}}\} denote a CRS dataset with N_{\text{dial}} dialogues. Each dialogue D consists of n_{\text{utt}} utterances denoted by D=\{u_{t}\}_{t=1}^{n_{\text{utt}}} , where u_{t} represents the utterance at the t -th turn and each utterance u_{t}=\{w_{j}\}_{j=1}^{n} contains a sequence of n words. The task of CRS is to generate the response and recommend desirable items based on the given dialogue history and item database. To be specific, given the dialogue history up to the t -th turn D_{t}=\{u_{i}\}_{i=1}^{t-1} and the item database \bm{\mathcal{I}} , the CRS needs to recommend a set of candidate items \bm{\mathcal{I}}_{t} from \bm{\mathcal{I}} , and generate the response u_{t} which includes the items \bm{\mathcal{I}}_{t} . The recommended candidate items set \bm{\mathcal{I}}_{t} could be empty when no recommendation is needed, or contain one or more items depending on the responses.

In this work, we apply our method to the movie recommendation ( i.e., \bm{\mathcal{I}} denotes a movie items set), but the process would be identical with other types of items. We follow prior work Wang et al. ( 2022c ); Yang et al. ( 2022 ) to adjust data samples and predict response with a single recommended movie per utterance .
Model Input
In PECRS, items are represented by their textual descriptions, hence both input streams are modeled as text. Nevertheless, we design a few special tokens to distinguish the various elements in PECRS.
Special Tokens.
Our PECRS is built upon a pre-trained LM under the decoder-only style, parameterized by \theta (e.g, GPT-2). However, LMs generally do not have the capacity for recommendation task. Thus, we define four special tokens, i.e., “[ITEM]”, “[SEP]”, “[REC]” and “[REC_END]”, and add them into the LM’s vocabulary to guide the model’s understanding of recommended items.
Item Metadata.
Prior work Zhou et al. ( 2020a ); Zhang et al. ( 2022 ); Wang et al. ( 2022a ); Zhou et al. ( 2022 ); Wang et al. ( 2022c ) usually exploits external KG to encode item representations. They generally regard items as nodes and model relations among items through R-GCN Schlichtkrull et al. ( 2018 ) , but neglect the rich textual descriptions of the items. In contrast, similar to Yang et al. ( 2022 ) , we explore to use the static textual metadata of items. Item descriptions can be fed into a language model directly, hence bypassing the semantic misalignment issue. To be specific, each item I_{j} is represented by affluent relevant information of the item rather than just its title. For movie recommendation, we use the following format “ Movie title [SEP] Actors [SEP] Director(s) [SEP] Genre(s) [SEP] Plot ” to describe a movie item, where [SEP] is used to mark the separation among different fields. Note this process can be directly generalized to other domains by using the meta information of items in the target domain.
Formally, let I_{j}=\{c_{j,k}\}_{k=1}^{l} denotes the j -th item textual data with l tokens, its output from LM is \bm{I}_{j}=[\bm{c}_{j,1},\ldots,\bm{c}_{j,l}] . We further adopt a MLP layer h_{\text{item}} with learnable pooling weight \bm{w} to aggregate the item representation as:

\bm{v}_{j}=h_{item}(\bm{w}^{T}\cdot\bm{C}_{j})
Dialogue Context.
The dialogue context is made of all utterances up to the current t -th utterance: D_{t}=\{u_{i}\}_{i=1}^{t-1} . The word embeddings of the i -th utterance are denoted as \bm{u}_{i}=[\bm{c}_{i,1},\dots,\bm{c}_{i,n}] . If any utterance u_{i} contains an item, it will be replaced by “[ITEM]” token and its item representation is also concatenated to the left side of the utterance’s word embeddings. Otherwise, it remains unchanged. Let \bm{v}_{\text{sep}} , \bm{v}_{\text{rec}} and \bm{v}_{\text{rec\_end}} denote the token representations of “[SEP]”, “[REC]” and “[REC_END]”, respectively. Suppose the i -th utterance contains an item, if it is from seeker , its token embeddings are represented as \tilde{\bm{u}}_{i}=[\bm{v}_{\text{sep}},\bm{v}_{j},\bm{v}_{\text{sep}},\bm{u}_{i}] ; if it is from recommender , its token embeddings are \tilde{\bm{u}}_{i}=[\bm{v}_{\text{rec}},\bm{v}_{j},\bm{v}_{\text{rec\_end}},\bm{u}_{i}] . Thus, the token embedding sequences of dialogue context are the concatenation of all utterances with \bm{v}_{\text{rec}} representation:

\bm{D}_{t}=[\bar{\bm{u}}_{1},\ldots,\bar{\bm{u}}_{t-1},\bm{v}_{rec}], (2)

where \bar{\bm{u}}_{i}=\tilde{\bm{u}}_{i} if the utterance contains items, otherwise \bar{\bm{u}}_{i}=\bm{u}_{i} .
Recommendation
The recommendation module contains two processes: retrieval and re-ranking. The retrieval process is to select candidate items relevant to dialogue context from item database. The re-ranking process further re-ranks the selected candidate items after aggregating knowledge from the dialogue context.
Retrieval.
We use the movie item in the response to be predicted as the ground-truth item, and sample M negative items from item database. Then, we use their textual descriptions to encode item representations via Equation 1 and derive ground-truth item \bm{v}_{p} and negative items \{\bm{v}^{\prime}_{j}\}_{j=1}^{M} .
As the dialogue context is ended with “[REC]” token (ref. Equation 2 ) and decoder-only LM can aggregate all contextual information via causal self-attention, we utilize LM’s output of “[REC]” token, denoted as \bm{d}_{t} , to represent query representation of dialogue context.
We adopt a noise-contrastive estimation (NCE) Gutmann and Hyvärinen ( 2012 ); Mnih and Teh ( 2012 ); Mnih and Kavukcuoglu ( 2013 ) objective to bring together the query \bm{d}_{t} with the positive key \bm{v}_{p} and push apart M negative (query, key) pairs formed by the set \bm{\mathcal{N}}=\{(\bm{d}_{t},\bm{v}^{\prime}_{j})\}_{j=1}^{M} .
The NCE objective is written as:

\mathcal{E}_{D_{t}}=\frac{e^{f(\bm{d}_{t})^{\top}\odot\bm{v}_{p}}}{e^{f(\bm{d}_{t})^{\top}\odot\bm{v}_{p}}+\sum\limits_{(\bm{d}_{t},\bm{v}^{\prime}_{j})\sim\bm{\mathcal{N}}}e^{f(\bm{d}_{t})^{\top}\odot\bm{v}^{\prime}_{j}}}, (3)

where f is a projection head with two-layer MLP and ReLU activation; \odot denotes the angular distance, \sqrt{2(1-\cos(\bm{a},\bm{b}))} , which measures the similarity between two vectors, \bm{a} and \bm{b} . The recall loss for retrieval process is defined as:

\mathcal{L}_{recall}=-\frac{1}{|\bm{\mathcal{D}}|}\sum\limits_{D_{t}\in\bm{\mathcal{D}}}\log(\mathcal{E}_{D_{t}})

Note we stop the gradients of LM and only optimize the pooling and MLP layers for item representations encoding during training (ref. Figure 2 ) to accelerate the learning process. The item representations will be reused in re-ranking process and the LM will be optimized at this stage accordingly.
Re-ranking.
The item representations derived from retrieval process are reused in the re-ranking process to aggregate the knowledge of dialogue context. To be specific, given both positive and negative items, we concatenate them with the token embeddings of dialogue context as [\bm{D}_{t},\bm{v}_{p},\bm{v}^{\prime}_{1},\ldots,\bm{v}^{\prime}_{M}] and feed into LM then MLP f to compute the context-aware item representations [\bm{q}_{p},\bm{q}_{1},\ldots,\bm{q}_{M}] . Note that we adopt a special attention mask to enforce that each item \bm{v}_{j} only attends to tokens from \bm{D}_{t} , and positional embeddings are removed for item tokens to avoid any position leakage. Then another MLP layer g is applied to compute the final item scores as \bm{r}=[r_{p},r_{1},\ldots,r_{M}] .
The training objective of re-ranking process is:

\mathcal{L}_{rerank}=\frac{1}{|\bm{\mathcal{D}}|}\sum\limits_{D_{t}\in\bm{\mathcal{D}}}f_{XE}(\bm{r},\bm{Y}), (5)

where \bm{Y}=[1,0,\dots,0] and f_{\text{XE}} denotes cross-entropy loss. Note we shuffle r and \bm{Y} jointly to avoid the positional bias of ground-truth labels.
If a data point has no recommended item in the response, we set \mathcal{L}_{\text{recall}}=\mathcal{L}_{\text{rerank}}=0 .
Response Generation
The response generation aims to predict the current utterance u_{t}=\{w_{j}\}_{j=1}^{n} by giving the dialogue context. During training, if the u_{t} contains an item to be recommended, the representations of the ground-truth item is appended to the corresponding dialogue context to guarantee that the LM generates the response relevant to the item. Then, the input for response generation is:

\tilde{\bm{D}}_{t}=[\bar{\bm{u}}_{1},\ldots,\bar{\bm{u}}_{t-1},\bm{v}_{rec},\bm{v}_{p},\bm{v}_{rec\_end}]

Otherwise, the input for response generation stays as \tilde{\bm{D}_{t}}=[\bar{\bm{u}}_{1},\ldots,\bar{\bm{u}}_{t-1}] . In general, the response generation is optimized by the standard next-token prediction objective as:

\mathcal{L}_{gen}=-\frac{1}{|\bm{\mathcal{D}}|}\sum\limits_{D_{t}\in\bm{\mathcal{D}}}\frac{1}{n}\sum\limits_{j=1}^{n}\log(p_{\theta}(w_{j}|w_{1:(j-1)},\tilde{\bm{D}}_{t})
Parameter-Efficient Learning
We exploit parameter-efficient fine-tuning (PEFT) techniques for training. PEFT can achieve comparable performance to standard fine-tuning Hu et al. ( 2023 ) with higher training efficiency and avoid the catastrophic forgetting issue of LM. Specifically, we leverage the LoRA Hu et al. ( 2022b ) method, which incorporates low-rank weight matrices into transformer layers to adapt LM to downstream tasks by fine-tuning the injected weights only.
In addition to LoRA layers, we also fine-tune the task-specific MLP layers f , g and h_{\text{item}} and the token embeddings of the four special tokens. PECRS only updates a small proportion (around 5\% ) of the total number of parameters in the model.
Training and Inference
The PECRS is trained in a singe-stage end-to-end manner by minimizing the following loss:

\mathcal{L}=\alpha\times\mathcal{L}_{recall}+\beta\times\mathcal{L}_{rerank}+\gamma\times\mathcal{L}_{gen}, (8)

where \alpha , \beta and \gamma are hyperparameters to balance the three losses. During training, we randomly sample M_{\text{train}} negative items and share them for computing the \mathcal{L}_{\text{recall}} and \mathcal{L}_{\text{rerank}} losses. Besides, we share the negative samples across batch elements and ensure that none of them is a positive for the dialogue contexts within a batch.

During inference, we first use PLM to encode the representations of all items in the database, which are reused for all dialogue contexts. Then the top- M_{\text{inference}} items with highest similarities to the dialogue context query are retrieved via f(\bm{d}_{t})^{\top}\odot\bm{v}_{j} (see Equation 3 ). We further re-rank the M_{\text{inference}} items to obtain the top-1 item as the recommendation output. In practice, we set M_{\text{train}}<M_{\text{inference}} . We show that M yields an important trade-off between efficiency and recommendation performance both during training and inference in Section 5.2 .
Moreover, the predicted item is appended at the end of the dialogue context rather than the ground truth in Equation 6 in order to prompt the model for response generation. To determine whether a movie should be recommended at inference, we check whether the “[ITEM]” token is present in the generated response.
Datasets.
We conduct experiments on two commonly used datasets, i.e., ReDial Li et al. ( 2018 ) and INSPIRED Hayati et al. ( 2020 ) . ReDial 1 1 1 https://redialdata.github.io/website/ contains 11,348 conversations ( 10,006 for train and 1,342 for test) about movie recommendation between seeker and recommender , which is constructed through crowd-sourcing workers on Amazon Mechanical Turk. INSPIRED 2 2 2 https://github.com/sweetpeach/Inspired is also about movie recommendation with smaller size of 999 ( 801 for train, 99 for development and 99 for test) and more flexibility given to workers. The statistics of both datasets are summarized in Table 1 .
Evaluation Metrics.
We follow the common practices Yang et al. ( 2022 ); Wang et al. ( 2022c ) to evaluate PECRS on both recommendation performance and response generation quality. For recommendation subtask, we measure recall with Recall@K (R@K) metric, taking K\in\{1,10,50\} .
In order to assess the recommendation coverage, we also report the number of different items predicted by the model over the test set, denoted as Unique . ReDial and INSPIRED contain 6,637 and 1,546 unique items in total ( Table 1 ) and 1,872 and 264 items in the test set, respectively.

We use both reference-based and reference-free metrics to evaluate response generation quality. For reference-based metrics,
we adopt ROUGE@K (RG-K) Lin ( 2004 ) with K\in\{1,2\} .
To verify whether the model could correctly predict a movie in response when required, we inspect the presence of the “[ITEM]” token in generated responses w.r.t. ground truth requirement of movie prediction via F-1 score. For reference-free metrics, we use Perplexity (PPL) to assess the text fluency and Distinct@K (Dist@K) with K\in\{2,3,4\} to measure the diversity of generated responses.
Implementation.
We choose GPT-2 Radford et al. ( 2019 ) as the backbone LM, and experiment with two different model sizes, i.e., GPT-2 small and GPT-2 medium, which enable us to compare against popular CRS approaches. Accordingly, we have PECRS-small and PECRS-medium .
We highlight that PECRS is flexible and can support other choices of decoder-only LMs. We use the public pre-trained checkpoints from HuggingFace transformers library Wolf et al. ( 2020 ) . We set M_{\text{train}}=150 for training and M_{\text{infer}}=700 for inference. For ReDial, we train for 10 epochs with effective batch size 8 ; while for INSPIRED, we train for 20 epochs with an effective batch size of 2 . Parameter optimization is performed by AdamW Loshchilov and Hutter ( 2019 ) with linear learning rate warmup strategy. We set maximum learning rate as 3e-5 for PECRS-small and PECRS-medium and warmup for 1 epoch. During training, we balance losses with \alpha=0.15 , \beta=0.85 , and \gamma=1.0 . We cap dialogue context length at 256 tokens and response length at 64 tokens.
We use checkpoint with the highest mean of R@1, R@10 and R@50 for inference.
PECRS generates the response with top-k sampling, using k=50 .
The movie item metadata is obtained from The Movie Database through tmdbv3api library 3 3 3 https://github.com/AnthonyBloomer/tmdbv3api .
Comparison with State-of-the-Art
The results on recommendation task are summarized in Table 2 . Note that RevCore Lu et al. ( 2021 ) and C 2 CRS Zhou et al. ( 2022 ) are not directly comparable to our method as they use additional movie review information. PECRS generally outperforms the baselines using KG and extra model, such as KGSF Zhou et al. ( 2020a ) and UniCRS Wang et al. ( 2022c ) , on both datasets. Compared to the baselines with single training stage, PECRS surpasses BARCOR Wang et al. ( 2022b ) and RecInDial Wang et al. ( 2022a ) . MESE Yang et al. ( 2022 ) also uses the item descriptions and employs two additional modules to encode items. In contrast, our PECRS is simpler and more straightforward, and it is the first approach without using either KG or supplementary module, but only relying on the pre-trained LM. PECRS-medium outperforms MESE for Recall@1 on ReDial, achieving SOTA, and largely surpasses MESE for all metrics on INSPIRED.
Besides, PECRS-medium is superior to -small on all metrics, which demonstrates that fine-tuning a larger LM brings more gains thanks to its stronger representation ability.

Table 3 summarizes the results on conversation task, where PECRS achieves promising performance on both types of metrics. Both PECRS-small and -medium surpass all baselines over Dist@3 and Dist@4. Comparing PECRS-small and -medium shows that Dist@K improvements can be achieved by scaling up the backbone model. Thus, we believe that larger LMs can bring better results, and fine-tuning them with plugin style to acquire CRS capability is a promising research direction. A human evaluation ( Table 4 ) for fluency and relevancy on ReDial test set with three volunteer graduate students with professional English proficiency confirms a preference for PECRS-small generated text over MESE outputs.
Ablation Study
We also conduct ablative experiments to analyze the architecture and optimization design of PECRS.
Reported in Table 5 , all the components and training strategies contribute to the performance gains on both recommendation and conversation tasks. In particular, recommendation collapses without either loss from its two-stage processes, i.e., retrieval and re-ranking ; and suffers without the generation loss. Sharing negative samples across batch elements and tasks leads to significant improvements on training efficiency and marginal gains on recommendation performance.

In Table 6 , we conduct a further ablation on the textual fields within items description. We observe that every field contributes to the recommendation performance, especially the plot. This suggests that richer metadata would yield even more recall gains.
Comparison with Large Language Models
Lastly, we compare our fine-tuning approach with Large Language Models (LLMs). Instruction-tuned LLMs have brought a seismic shift in NLP recently, due to their ability to seamlessly conduct many tasks in a zero-shot fashion through prompts, by-passing the need for task-specific supervised fine-tuning (Sanh et al., 2021 ; Wei et al., 2021 ; Ouyang et al., 2022 ) , including in recommender systems (Hou et al., 2023 ) .

We use two popular LLMs: Llama-2-7B-chat 4 4 4 https://huggingface.co/meta-llama/Llama-2-7b-chat-hf (Touvron et al., 2023b ) , and Vicuna-1.5-7B 5 5 5 https://huggingface.co/lmsys/vicuna-7b-v1.5 (Chiang et al., 2023 ) . For each model, we condition on the context, and prompt the LLM to predict the Recommender response, which should include a movie name. We infer in bfloat16 , decode with greedy decoding, and check if the ground-truth movie name is included in the generated response. As seen in Table 7 , the conversational recommendation capability of LLMs in zero-shot is very promising, as they outperform PECRS-small in Recall@1 on INSPIRED. However, due to the lack of a dedicated recommendation module, LLMs used in this fashion cannot suggest a full list of items, hence their recall plateaus at the Recall@1 value. They also tend to recommend fewer different movies (lower Unique ). Exploring the ranking of a larger list of recommended items with LLMs is a promising future research avenue.
Analysis
In this section, we provide more detailed insights about the behavior of PECRS.
Conversation Evaluation
We first study the effects of different LM’s decoding strategies on conversational performance over Dist@K metric. Specifically, we analyze the greedy decoding, beam search, diverse beam search Vijayakumar et al. ( 2018 ) , top-k sampling Fan et al. ( 2018 ) and nucleus sampling Holtzman et al. ( 2020 ) strategies on PECRS-small. Reported in Table 8 , reference-based metrics (RG-K) show much less variance on different decoding strategies compared to the reference-free metrics (Dist@K). Meanwhile, the correlation between reference-based and reference-free metrics is weak under different decoding strategies. Moreover, PECRS without training for generation can achieve 11.907 on Dist@2 metric (see w/o Generation loss in Table 5 ), but merely 7.76 on RG-1 metric. This observation implies that Dist@K metrics are not reliable to evaluate the quality of response generation . Since Dist@K metrics have become the most popular choice in evaluating conversation performance of CRS Zhou et al. ( 2022 ); Wang et al. ( 2022c ); Yang et al. ( 2022 ) , we advocate for applying other metrics, in particular reference-based metrics including n-gram overlap like ROUGE or semantic similarity like BERTScore Zhang et al. ( 2019 ) , to provide more accurate evaluation on the response generation of CRS.
Negative Sampling
Now we analyze how the hyper-parameters of negative sampling, i.e., M_{\text{train}} and M_{\text{inference}} , affect the recommendation performance. Figure 3 illustrates the results of different choices of M_{\text{train}} and M_{\text{inference}} pairs. In general, M_{\text{train}} and M_{\text{inference}} have significant impacts on the recommendation performance, and larger M_{\text{train}} and M_{\text{inference}} lead to better results. However, increasing M will reduce the training and inference efficiency. Thus, there is a trade-off between efficiency and recommendation performance for the selection of M .
Conversation Turns
Lastly, we investigate how robust is PECRS with regards to the richness of dialogue context. In Figure 4 , we group data points by number of utterances happening before the CRS response.
We observe that PECRS performs well in recommendation for a wide range of context length, with only a moderate drop when there is only one prior utterance.
Conclusion
In this work, we formulate conversational recommendation as a language processing task and propose a unified parameter-efficient CRS (PECRS) framework to solve it in a single-stage end-to-end manner. PECRS effectively addresses the inferior training efficiency via parameter-efficient fine-tuning techniques and semantic misalignment issues via joint conversation and recommendation modeling. Through experiments, we show that PECRS achieves performance competitive with SOTA on both recommendation and response generation on benchmark datasets.
Moreover, for response evaluation, we reveal the commonly used Dist@K metrics are not reliable, and advocate for reference-based metrics (e.g ROUGE) for more accurate evaluation. Generally, we show that it is promising to explore unified framework for CRS under the natural language paradigm via language model and rich textual items data.
Limitations
Our work adheres to standard practices for dataset construction and model evaluation. However, we acknowledge three limitations: (1) Recommender utterances containing multiple items are separated into individual data points, which is sub-optimal as the model may only be accurate for the top-ranked item in each data point. (2) If we train PECRS to predict multiple items within the same utterance, it is challenging to compare with current methods, as they do not make simultaneous predictions. (3) All items mentioned by the recommender are considered recommendations, although some may be references to previous discussions or express dislikes rather than actual recommendations.

The maximum context length for the backbone LM is another limitation. We have demonstrated that increasing M_{\text{inference}} yields better recommendation performance (ref. Section 5.2 ). However, we are constrained by the maximum input length of 1024 for GPT-2, which limits the candidate set size after concatenating with dialogue context. The potential extensions may involve performing inference with multiple forward passes to score batches of M_{\text{inference}} items, or using a backbone that supports longer input lengths, albeit at a higher computational cost.
We only experiment with relatively small backbone, i.e., GPT2-small and -medium, due to resource limitation. However, PECRS is flexible and can be seamlessly applied to larger backbones like LLaMA Touvron et al. ( 2023a ) .
Figure 1: An example of dialogue from ReDial Li et al. ( 2018 ) , where blue color denotes the movie items.; An example of dialogue from ReDial Li et al. ( 2018 ) , where blue color denotes the movie items.