Rhapsody: A Dataset for Highlight Detection in Podcasts
Authors: Younghan Park
Anuj Diwan
David Harwath
Eunsol Choi
Venue: COLM 2025 (2025)
Keywords: podcast highlight detection
long-context reasoning
spoken language processing
TL;DR: We present a dataset of 13K podcast episodes paired with segment-level highlight scores.

Abstract: Podcasts have become daily companions for half a billion users. Given the enormous amount of podcast content available, highlights provide a valuable signal that helps viewers get the gist of an episode and decide if they want to invest in listening to it in its entirety. However, identifying highlights automatically is challenging due to the unstructured and long-form nature of the content. We introduce Rhapsody, a dataset of 13K podcast episodes paired with segment-level highlight scores derived from YouTube's 'most replayed' feature. We frame the podcast highlight detection as a segment-level binary classification task. We explore various baseline approaches, including zero-shot prompting of language models and lightweight fine-tuned language models using segment-level classification heads. Our experimental results indicate that even state-of-the-art language models like GPT-4o and Gemini struggle with this task, while models fine-tuned with in-domain data significantly outperform their zero-shot performance. The fine-tuned model benefits from leveraging both speech signal features and transcripts. These findings highlight the challenges for fine-grained information access in long-form spoken media.

Published as a conference paper at COLM 2025
RHAPSODY: A Dataset for Highlight Detection in Podcasts
Younghan Park', Anuj Diwan', David Harwath Eunsol Choi
'Yonsei University, "The University of Texas Austin, New York University younghanpark@yonsei.. www.nnneeeeararr....... kr, {anuj. diwaj, Srrssth@eutexa.cccm eunsolênyu. edu
Abstract
Podcasts have become daily companions for half a billion users. Given the enormous amount of podcast content available, highlights provide a valuable signal that helps viewers get the gist of an episode and decide if they want to invest in listening to in its entirety. However, identify- ing highlights automatically is challenging due to the unstructured and long-form nature of the content. We introduce RHAPSODY, dataset of 13K podcast episodes paired with segment-level highlight scores derived from YouTube's 'most replayed' feature. We frame the podcast highlight detection segment-level binary classification task. We explore various baseline approaches, including zero-shot prompting of language models and lightweight fine-tuned language models using segment-level classifica- tion heads. Our experimental results indicate that even state-of-ther of-the-art lan- guage models like GPT-4o and Gemini struggle with this task, while models fine-tuned with in-domain data significantly outperform their zero-shot performance. The fine-tuned model benefits from leveraging both speech signal features and transcripts. These findings highlight the challenges for fine-grained information access in long-form spoken media. We release our codes and dataset at https: / //gi thub. com/ comm///ounggaatttkk/rapssoooo
Introduction
Podcast consumption has grown significantly in recent years, prompting research into content-related tasks such as search (Tian et al., 2022), recommendation (Fan et al., 2023; Nadai et al., 2024), and summarization (Jones et al., 2020; Karlgren et al., 2021). However, identifying the most interesting parts (i.e., highlights) of podcast episodes remains relatively underexplored. This is an important problem, as the predicted highlights could help listeners quickly decide whether an episode is worth their time. Despite its various use cases, progress in this area has been limited due to several challenges: podcasts are often lengthy, and the most interesting parts of the podcast are subjective, making it challenging to gather coherent annotations.
In this paper, we present the first study on automatically locating highlights of podcast episodes. Our approach is inspired by the 'most replayed' feature of YouTube: a graph displaying frequently replayed parts of videos (referred to the replay graph) (YouTube) 2022). This feature, which aggregates the views of thousands of users, opened doors for researchers to study video understanding tasks (Duico et al., 2023; Sul et al., 2023; Kim et al., 2025). Compared to such prior work which mainly consider visual features, we focus on the spoken content.
We first propose the podcast highlight detection task, and derive RHAPSODY, large-scale dataset (containing 13K podcast episodes from various domains), visualized in Figure 1. This task long-context in nature because the average podcast episode in our dataset is about 30 minutes long and contains an average of 5K words. Each podcast is divided into 100 equal-length segments, of which a subset (with an average of segments) is identified
Work was partly done while at UT Austin. Equal advising.
RHHAPOdY: Replay graph, Highlights, Podcasts, Segment-level, Detection.

---

Published as a conference paper at COLM 2025
(1) Creating Rhapsody
Podcast Audio
[1| [1-1111111
Highlight Labels
X
Convert
x 13K Podcasts
Replay Graph
0.5
(2) Benchmarking on Rhapsody
Ground-Truth [2, .98
Prediction P = (1, 98, 100)
Podcast Audio
Audio Segments
Audio Features
Split
Extract
1
+|p+ 4|4 .. x100
x100
ASR
Models
Split
Transcript
Transcript Segments
Segment Summaries
Figure 1: (Upper) RHAPSODY dataset derived from 13K podcast episodes from YouTube. The audio stream of an episode split and paired with each data point in the replay graph of the video. (Lower) A model is asked to predict the indices of highlight segments, given the preprocessed textual and audio features of each episode segment. Ground-truth (GT) highlights are defined as indices of peaks in the replay graph (Section 2.2) and compared to the predicted indices for benchmarking.
as being the most-replayed highlights. We convert the replay graph into the subset of these most-replayed segments. Given input features from the podcast episode, such as episode transcript, episode title, or audio features, model must identify this subset of most-replayed segments.
To tackle this new task, we propose suite of baseline systems. We first present text-based BM25 (Robertson & Zaragoza, 2009) approach, which treats highlight detection as the task of retrieving the segments whose transcript summaries (as summarized by off-the-shelf LLM) have the highest BM25 score to the episode title. We next present zero-shot prompting results for 2024). Lastly, we present learned models on this task, which uses hidden representation from fine-tuned LLM and optionally audio features. This model outperforms GPT-4o's and Gemini's zero-shot performance by large margin. Yet, the task remains very challenging, with maximum hit rate of 49.0% with our best approach.
state-of-the-art large language models (OpenAi, 2024a; Google Deepmind,
We sum up our contributions as follows:
We formally describe the podcast highlight detection task, explain how supervision signals can be generated from replay graphs, and propose a set of automatic evaluation metrics for this task.
We introduce RHAPSODY, a dataset of 13K podcast episodes from various domains, containing viewer replay scores for each segment.
We benchmark the performance of various methods, including zero-shot LLM prompting and our fine-tuned language models, across text-only and text-and-audit settings.
We release all our data and code support future research podcast understanding. Given the limitations of current models, our benchmark can support rich future work, including methods that combine text and audio signals and methods that focus the long-context ability of LLMs.
2 RHAPSODY: A Dataset for Highlight Detection in Podcasts
2.1 Task Formulation
YouTube provides replay graphs consisting of 100 time-aligned replay scores that display the most frequently replayed parts for most videos that have sufficiently high view counts
2

---

Published as a conference paper at COLM 2025
20 0=0.0
K= 20, t=0.5
0.75
0.50
0.25
0.00
1.00
0.75
0.50
0.25
0.00
100
20
100
Figure 2: Comparing two settings for defining GT highlights indices G. (Left) Naively defining segments with top-K replay scores highlights. (Right) Introducing 7, hyper- parameter to impose minimum threshold (dotted green lines). The light blue lines at the beginning of the graphs represent the graph before bias correction (Appendix A.3).
(typically 2 50K views). We denote A as the audio track of a podcast episode with total duration and M as the replay graph of A: M [m, 100 where m; E [0,1]. We (i-iid
divide the audio track A into 100 equal-length segments c, TW second to iid second, treating each Mi as representing the replay score of its corresponding audio segment c, As described in Section 2.2, using the replay graph M, we derive set of Ground-Truth (GT)
highlight segment indices G = } (where 8j {1,2,.... 100}) that correspond to the most-replayed segments in the podcast episode.
We propose a podcast highlight detection task where a model is provided input features derived from the podcast episode and is asked to predict a set of highlight indices which is compared to the GT highlight segment indices G. We consider both episode-level input features, such as the episode title, and segment- level input features, such as transcript summaries or audio features for each segment.
{p&] Np (where Pk E .1,
2.2 Data Collection Process
Data Sources scrape the list of most popular podcast creators from YouTube,? yielding 100 channel IDs. Using YouTube Data API v3, we then collect metadata for these channels and their playlists. We discard playlists that (1) are not explicitly set as podcasts by their creators and (2) are not in English. We further filter out playlists that predominantly feature visual content, such music videos, TV series reviews, or reaction videos. We also discard videos for which the replay graph is unavailable. Then, we limit the maximum duration of the episode to an hour to prevent single data point replay graph corresponding long span of the audio. Moreover, we exclude episodes for which automatic transcription has more than 10 segments without any detected words. Details on filtering channels and playlists are in Appendix A.1. We download the audio track, replay graph, and YouTube video metadata for each podcast episode. The metadata includes episode-leve features such the title, creator-written description, video categories, and view statistics.
Converting Replay Graph into Set of GT Highlight Indices Typically, only small number of segments for each podcast have high replay scores. We convert the replay graph into set of GT indices
{g}} Nc for evaluation. The simplest approach for defining Gs to set it as the indices of data points of the top-K replay score (Duico et al., 2023; Sul et al., 2023). However, this approach causes false positive highlights when the replay graph is relatively flat. Instead, we introduce a hyperparameter T representing minimum replay score for being considered a highlight segment. Specifically, we label at most indices as GT among a set of candidate indices (i: m; > - We opt to use and T = 0.5 for
httttp:::/www..uuue com/ppodcasts/poplaarccr Phttps://deve www.:oppp.....oolle.eeeeooooo..e com/youtube///
3

---

Published as a conference paper at COLM 2025
Category
episodes
# channels
avg. duration (min)
# words (K)
# views (M)
Entertainment
4,873
36
28.02 (15.49)
4.95 (2.82)
0.59 (1.32)
News & Politics
4,154
16
24.65 (16.38)
4.11 (2.81)
0.47 (0.65)
Comedy
632
15
25.50 (17.15)
4.83 (3.33)
0.29 (0.70)
Science & Tech. Education
623
5
33.57 (18.80)
5.52 (3.23)
0.50 (0.96)
591
12
26.58 (17.34)
4.37 (3.24)
0.92 (1.79)
Sports
496
11
29.58 (19.18)
5.36 (3.66)
0.27 (0.70)
People & Blogs
474
24
33.37 (20.47)
6.05 (3.87)
0.68 (1.13)
Unknown
1,317
66
27.04 (16.23)
4.66 (2.90)
0.63 (1.29)
13,364
90
27.21 (16.70)
4.71 (3.01)
0.55 (1.13)
Table 1: Statistics of podcast episodes in RHAPSODY. We report the number of podcast episodes and channels for each category, omitting categories with less than 100 episodes.. We report the average duration of an episode, the number of words in the transcript, and the view count of the episode. We report the standard deviation parentheses.
all experiments based defining highlight segments. We provide more examples in Appendix A.2.
manual data inspection. Figure compares two settings for
The replay graphs exhibit biases towards the first segment, likely as users frequently replay the beginning of the content without finishing it. In the original form, 75% of the episodes in the entire dataset have one of the GT highlights setting the first value of the graph equal to the second one, preventing models from learning noisy patterns. Appendix A.3 provides more detailed explanation of bias correction.
the first segment. We correct this bias by
Dataset Statistics Our dataset includes 13,364 examples, each corresponding to a single podcast episode. We randomly split our dataset into the train, validation, and test sets with 70:10:20 ratio. Table provides an overview of the statistics of podcast episodes.
3 Approaches for Predicting Podcast Highlights
We compare the performance of nine systems: random sampling and frequency-based baselines, a BM25-based method, three zero-shot prompting methods with GPT-4o and Gemini, and three fine-tuned LLMs-one with segment-level binary classification head and the others additionally incorporating two types of audio features. We first describe feature extraction process from raw audio that will be used by systems.
3.1 Input Data Preprocessing
Transcript Segmentation To map the podcast transcript to the replay graph, which consists of 100 numbers, we split the transcript into 100 segments. We first run the podcast audio through WhisperX (Bain et al., 2023), a version of Whisper (Radford et al., 2022) that outputs time-aligned ASR transcription providing word-level alignments of the form (w; } where wi represents the jth word, Sj its start time, ej its end time, and Na the total number of words. Given this WhisperX output, we segment the transcript into 100 equal-duration intervals based on the episode segments [c word wy is assigned segment if both sj and e; fall entirely within that segment's time span. If word spans two segments, it is assigned to the segment with the greater overlap. We use shisr-lllarge-- for the transcription model and WAV2VEC2 ASR LARGE LV60K 960H (Baevski et al., 2020) for the alignment model.
Summarization of Transcript Segments We summarize the transcript segments using 1i0---mmii--22-0-777 -8-88-07-18 (OpenAl, 2024b) and use the summary output as input textual features to the model. We prompt LLM to process 10 consecutive segments at time while providing the previous context to avoid providing an extremely long prompt. We provide more detailed information along with pseudocode in Appendix B.1.

---

Published as a conference paper at COLM 2025
Audio Feature Extraction To capture emotional characteristics in podcasts, explore two types audio features sattess-ddoiaaeee-llee--eee
(DVA) embeddings and HuBERT features. The DVA model represents emotions along three cimensions-ddmmnnncee valence, and arousal providdng a structured framework for analyzing affective states in speech (Russell & Mehrabian, 1977). To learn richer latent patterns, leverage the hidden repre- sentation from the final transformer layer of an off-the-shelf DVA-based emotion recognition model (Wagner et al., 2023). For HuBERT features, we take the hidden representations from the 10th layer of the model, and average-pool them over time. A more detailed explanation of audio feature extraction can be found in Appendix B.2.
3.2 Models
Random Sampling and Frequency-Based As heuristic baselines, consider random sampling and requeency-based baselines. The random sampling method samples unique elements from the population of 100 integers {1,2, 100). The trequency-based method precomputes the occurrence count of each index across all GT highlight indices in the training set and selects the top-K most frequent ones. In both methods, is sampled from the distribution of [G], the number of GT indices, in the training set. All results from these baselines were averaged over 1,000 independent experiments.
BM25 As text-only baseline, adapt BM25 for our task. We treat the summaries of each transcript segment as the corpus and the episode title as the query. We use 3,0 as the BM25 threshold, which is determined through manual tuning the validation set.
Zero-Shot Prompting with GPT-4o and Gemini To test state-of-the-art LLMs on this long- context reasoning task, we consider three zero-shot settings: GPT-4o (text-only), Gemini (text-only), and Gemini (text audio). For the GPT-4o (text-only) and Gemini (text-only) settings, we prompt 4pt-44-2024408-8 and gemini-2.. o-flash-001 with the title of the episode and transcript summaries for each segment, asking them to identify the segment indices that are most likely to be replayed. The model is instructed to rank answer segments in descending order of replay likelihood for proper computation of AP, and chain of-thou prompting is applied in the prompt. For the Gemini (text audio) setup, we also provide the full podcast audio. The text-only and the text + audio prompts can be found in Appendix C.2.
Fine-tuned LLMs with Segment-Level Classification Heads We fine-tune an LLM on RHAPSODY using parameter- efficient fine-tuning and replacing the de- fault next-token prediction head with custom segment-level binary classification head, shown in Figure 3. We develop three models, a text-only model that only uti- lizes textual features and two text audio models that utilize both textual and audio Ratures-DVa.
Estimated Likelihood for Each Segment
Linear
Optional
LLM
QLORA
Linear
HuBERT embeddings.
The input to the text-only model is 100 tran-
special end-of-segment tokens
VDA Embeddings
script summaries, one for each of the 100 segments, with end-of -f--o--segment tokens ap- pended after each segment. We extract one
text transcript tokens
ASR Segmenting VDA Model
LLM hidden representation for each of the 100 -nd-mm--egmeet tokens and apply a bi- nary classification head, consisting of lin- ear layer and a sigmoid activation, to obtain 100 probability values, each estimating the likelihood that the corresponding segment is highlight.
Podcast Audio all
Figure 3: The illustration of proposed ap- proach. Text and audio features are processed separately before being fused.
We freeze the base model, add QLoRA (Hu et al., 2022; Dettmers et al., 2023) adapters and optimize these adapters well as the classification head and linear layer of audio features
5

---

Published as a conference paper at COLM 2025
System
Hit Rate
Precision
Recall
F1
AP
Random Sampling
0.220
0.053
0.052
0.039
0.021
Frequency-based
0.224
0.072
0.064
0.051
0.031
BM25 (text-only)
0.216
0.051
0.060
0.04
0.024
GPT-4o (text-only)
0.265
0.088
0.053
0.057
0.032
Gemini (text-only)
0.327
0.082
0.074
0.065
0.037
Gemini (text audio)
0.293
0.090
0.065
0.063
0.040
Fine-tuned Llama (text-only)
0.452
0.202
0.178
0.163
0.134
Fine-tuned Llama (text DVA)
0.477
0.216
0.187
0.175
0.140
Fine-tuned Llama (text HuBERT)
0.490
0.266
0.197
0.197
0.153
Table 2: Experimental results test examples that have at least one ground-truth highlight
(GG) > 0; 2,467 episodes). We bold the best-performing method and underline the second- best-performing method. We set all the metrics to be zero when there no predicted indices (IP) = 0).
using binary cross-entropy loss. We use Llama-3. 3. 2-1--Irutt (Meta, 2024) as the base model' and apply A optimization with weight decay. We manually tuned hyperpa- rameters on the validation split and selected the configuration that achieved the highest average performance across all evaluation metrics we report. As detailed in Appendix D, the hyperparameters we tune include the batch size, LoRA hyperparameters, learning rate, and prediction thresholds. The resulting configurations yielded approximately 5.6M, 6.7M, and 6.2M trainable parameters for the text-only, text DVA, and text HuBERT settings, respectively.
4 Experiments
4.1 Evaluation Metrics
N₂ We aim to measure the similarity of sequence of predicted indices = { based on sequence of GT indices G {gj} NG j=1'
Since this setting can viewed as a document retrieval task where P is set of retrieved documents and G is set of relevant documents, we adopt five widely used evaluation metrics for document retrieval - hit rate, precision, recall, F1, and average precision (AP). When computing AP, we ensure mg: 2 mg; is satisfied
for all S < Ng.
The number of model predictions heavily influences many evaluation metrics, such as precision and recall. Therefore, we tune the prediction thresholds of the BM25 and fine-tuned Llama models on the validation set, aiming to match the average number of predictions across all methods. We ensure that methods for which we cannot control the number of predictions, such as zero-shot prompting of GPT-4o and Gemini, do not produce excessively many predictions, enable fair comparison with other models.
4.2 Results
We report the main experimental results in Table Our experimental results show that even the best-performing model achieves only 0.266 precision and 0.197 F1 score, highlighting the difficulty of podcast highlight detection and the room for improvement beyond current approaches. Yet, this is subjective task with derived gold labels, thus the upper-bound performance might not be perfect. We still anticipate a large room for improvement given the current performance level.
As expected, random sampling and frequency-based approaches performed the worst. Amongst the zero-shot prompting approaches, the Geminí model performed the best, while GPT-4o performed similarly to the frequency-based baseline, suggesting that it struggles
We <|reserved special taken 0|> for end-of-segment token.

---

Published as a conference paper at COLM 2025
Method
Ent.
News & Poli.
Comedy
Sci. & Tech.
Edu.
Sports
Ppl. Blogs
Random Sampling
0.240
0.157
0.238
0.227
0.227
0.257
0.211
Frequency-bused
0.235
0.141
0.400
0.153
0.220
0.326
0.259
BM25 (eextronyy)
0.297
GPT-4o (text-only)
0.286
0.173
0.203
0.275
0.237
0.360
0.264
Gemini (text-only)
0.326
0.219
0.316
0.405
0.326
0:432
0.297
Gemini (text audio)
0.315
0.185
0.233
0.351
0.319
0.360
0.253
Fine-tuned Laama (text-only)
0.489
0.384
0.714
0.313
0.326
0.586
0.527
Fine-tuned Llama (text DVA)
0.541
0.403
0.714
0.374
0.341
0.441
0.516
Fine-tuned Llama (lext HuBERT)
0.642
0.427
0.820
0.420
0.444
0.649
0.637
Table 3: Average hit rate across subsets of podcast episodes grouped by genre. When multiple categories are listed in the video metadata, the first category is used as the genre.
1.0
Frequency-based
1.0
Frequency-baset
4o (text only)
GPT (text only)
0.8
Gemini (text + audio)
0.8
Gemini (text + audio]
Precision
Liaraa (text + HuBERT)
Liama (text HuBERT)
0.6
Recall
0.6
0.4
0.4
0.2
0.2
0.0
0.0
10
20
6D
10
20
60
Duration (minutes, rounded up to nearest 10)
Duration (minutes, rounded up to nearest 10]
Figure 4: Precision (left) and recall (right) on test set, grouped by podcast duration. The shaded area represents #1 standard deviation around the mean.
to infer highlight segments from textual content without explicit training. Finally, our fine-tuned LMs with a segment-level classification head trained on in-domain data, despite having only 6.2M trainable parameters, significantly outperform all baselines across all metrics. However, the absolute performance remains lacking, emphasizing the need for further advancements in this task.
Effect of Audio Features Incorporating audio features alongside text leads to consistent performance improvements for the fine-tuned Llama model, suggesting that prosodic and acoustic cues provide valuable information for identifying highlights. Yet, the zero-shot Gemini result with audio does not consistently outperform the text-only Gemini result. This suggests that, while there are useful signals in the spoken content, fine-tuning with in-domain data might be needed to leverage such signals. Providing 100 segmented audio clips may contribute to improved zero-shot Gemini performance compared to providing the full episode as input.
Performance Different Genres and Duration Groups In Table 3, present the average hit rate depending on podcast genres. Fine-tuned Llama model with HuBERT features performs the best, regardless of podcast genre. Notably, fine-tuned Llama models generally outperform other models in the 'Comedy' genre by a huge gap. It is also interesting that the fine-tuned Llama model without audio signals performs better than the with DVA features in the 'Sports' genre. Additionally, LM-based methods don't perform as well in the 'News Politics' genre as the other genres.
In Figure 4, we group podcast episodes by rounding up the durations to the nearest 10 minutes.5 5 We do not observe a clear trend; most systems perform similarly regardless of podcast duration.
9Three L.I.-based methods with similar average number of predictions are plotted, along with the frequency-basee baseline. We report the same set of methods also in Figure 5 and Figure 6.
7

---

Published as a conference paper at COLM 2025
1.2
Frequency-based
1.0
Frequency-based
GPT-4o (text-only)
GPT-4o (text-onlyl
Gemini (text audio)
0.8
Gemini (text + audio]
0.8-
FT Liama (text HUBERT)
FT Liama (text HUBERT)
Precision
0.6
0.6
Recall
0.4
0.2
0.2
0.0
0.0
-0.2
10 [G]
12
14
16
18
20
10
12 14
16
18
20
[G]
Figure 5: Precision (left) and recall (right) test set, grouped by the number of GT high- lights. In |G] subset, we set both metrics the model prediction is an empty set and otherwise O. The shaded area represents i standard deviation around the mean.
14-
Ideal
System
# Highlights
12-
Frequency-based
(text-only)
Ground-Trin
5.32 (4.51)
10
Gemini (text + audic)
Random Sampling
5.17 (4.41)
HuBERT)
Frequency-
5.17 (4.41)
BM255
5.05 (5.36)
801-4o-0024-08-06
3.83 (1.16)
Gemini (text-unly)
6.03 (2.47)
Gemini (text audio)
4.32 (1.54)
FT Llama (text-only)
3.94 (3.64)
FT Llama (text DVA)
3.96 (3:46)
0:
5 10 10 th
FT Llama (text HuBERT)
(4.09)
Table 4: The average number of GT and predicted highlights per episode for each method. We report the stan- dard deviation in parentheses.
Figure 6: The average number of predicted high- lights ([Pl) for each subset grouped by the number GT highlights ([G)). The shaded area represents +1 standard deviation around the mean.
Performance Across Varying Number of GT Highlights To analyze model performance based on the number of GT highlights per episode, we evaluate precision and recall across subsets of episodes grouped by their G]. Figure shows that the fine-tuned Llama model consistently outperforms all the other methods in terms of precision and recall in most subsets.
Analysis the Number of GT and Predicted Highlights In Table 4, report the average number of predicted highlights ((P)) to ensure fair evaluation of results. The baseline methods, on average, produce higher number of predictions compared to all non-baseline methods. While zero-shot prompting with GPT-4o and Gemini (text + audio) produces lower average IP), their performance (as shown in Table 2) is superior the baseline methods. This indicates that the actual performance gap between them is larger than what the reported numbers indicate. Furthermore, our proposed method achieves significantly better performance than all other approaches, including zero-shot GPT-4o, while maintaining similar or lower average |P]. This supports our claim that our proposed approach is more effective.
Figure further illustrates how each model adjusts its number of predicted highlights. Ideally, a model should predict the same number of highlights as the GT (dotted line). However, baseline models predict roughly the same number of highlights regardless [G]. Notably, GPT-4o and Gemini also exhibit little variation in [P] across different G values, whereas our approach shows a clear upward trend, indicating better alignment with the GT distribution.
8

---

Published as a conference paper at COLM 2025
5 Related Work
Fine-Grained Podcast Information Access The podcast highlight detection task also falls under fine-grained information access tasks, as it focuses on identifying the most engaging segments within a podcast episode. Clifton et al. (2020) introduced 100,000 Podcasts, comprising nearly 60,000 hours of speech, along with their case study on spoken passage retrieval. Using this corpus, Jones et al. (2020) and Karlgren et al. (2021) further ran a track on podcast segment retrieval and received submissions from the public, while providing tasks, human annotations, and evaluation measures. Recently Ghazimatin et al. (2024) proposed PODTILE, which automatically generates chapters for a podcast episode, given its metadata and transcript. However, all of these works relied on human annotation, thus limited to smaller scale study.
Video Highlight Detection Highlight detection is query-agnostic and inherently subjec- tive task that requires multiple annotators. Due to the high cost of frame-wise or clip-wise annotation, earlier datasets are often small-scale (Sun et al., 2014; Song et al., 2015) or provide weak supervision (Gygli et al., 2016; Xiong et al., 2019). To address the need for large-scale datasets, Duico et al. (2023), Sul et al. (2023), and Kim et al. (2025) introduced the replay graphs for video highlight detection, showing that segments that are replayed many times align with human perception of highlights. Most works rely primarily visual features (Lin et al., 2023; Sun et al., 2024; Yang et al., 2024). While some methods incorporate multimodal features (Liu et al., 2022; Zhou et al., 2024), non-visual features contribute little, and audio features used, such as PANN (Kong et al., 2019), do not capture spoken content. To our knowledge, no prior work has explored time-variant textual features, such as transcripts, for highlight detection in streaming media.
Written and Spoken Passage Retrieval Traditional approaches rely on sparse represen- tation such as BM25, while recent advancements leverage dense embeddings from neural models (Karpukhin et al., 2020; Wu al., 2021; Yang et al., 2021) trained with contrastive learning objective. A few work also explores spoken passage retrieval (Lee et al., 2018; Lin al., 2024). The podcast highlight detection task can be viewed as a form of passage retrieval, where the corpus consists of segmented podcast episodes, and the query is the entire episode itself.
Conclusion
In this work, we introduce the task of podcast highlight detection and present RHAPSODY, dataset of 13K podcast episodes and binary highlight labels derived from YouTube's s replay graph feature. We benchmark the performance of different approaches with five automatic evaluation metrics. Our benchmark results demonstrate that even state-of-the-art LLMs, such as GPT-4o, struggle with this task, performing on par with simple heuristics. In contrast, our fine-tuned 1B Llama model significantly outperforms all the other approaches. However, the absolute performance of the best model remains low, underscoring the challenges of detecting meaningful highlights in podcast episodes.
Future work can explore improved modeling and inference methods, such as incorporating rich audio or contextual cues to further enhance podcast highlight detection. Our system relies on separate components for transcription, summarization, audio feature extraction, and highlight prediction, which may introduce compounding errors. Joint optimization in an end-to-end framework or improved input processing may mitigate this. Additionally, the structure of replay graphs itself and adaptive strategies for different podcast genres are another promising direction.
Acknowledgments
We thank all podcast creators on YouTube for creating interesting and valuable content. We also thank Youngjae Yu, Seungju Han, and Junhyeok Kim for their valuable input in the ideation of this work. Thanks to Xixi Hu for help with data collection.

---

Published as a conference paper at COLM 2025
Ethics Statement
The dataset we introduce is created from podcast episodes available on YouTube, and we respect the rights of their respective creator. Our use of this data is strictly for academic research purposes. Following previous works utilizing publicly available videos from YouTube (Zellers et al., 2021; Han al., 2023; Yang et al., 2023), we release only video IDs, replay graphs, and other processed features, while not releasing the original audio files. Additionally, we recognize that some podcasts in the dataset may contain sensitive content, including harmful, explicit, or discriminatory language. We encourage future research to consider ethical implications and mitigation of potential biases.
References
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli, wav2vec 2.0: A framework for self-supervised learning of speech representations. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Max Bain, Jaesung Huh, Tengda Han, and Andrew Zisserman. Whisperx: Time-accurate speech transcription of long-form audio. In Interspeech, 2023.
Ann Clifton, Sravana Reddy, Yongze Yu, Aasish Pappu, Rezvaneh Rezapour, Hamed Bonab, Maria Eskevich, Gareth Jones, Jussi Karlgren, Ben Carterette, and Rosie Jones. 100,000 podcasts: A spoken English document corpus. In International Conference on Computational Linguistics (COLING), 2020.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient fine- tuning of quantized lims. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Alessandro Duico, Ombretta Strafforello, and Jan van Gemert. Can we predict the most replayed data of video streaming platforms? In International Conference on Computer Vision (ICCV) Workshop on Al for Creative Video Ediling and Understanding Exlended Absiract Track, 2023.
Ziwei Fan, Alice Wang, and Zahra Nazari. Episodes discovery recommendation with multi-source augmentations. arviv preprint xrXiv:2301.01737, 2023.
Azin Ghazimatin, Ekaterina Garmash, Gustavo Penha, Kristen Sheets, Martin Achenbach, Oguz Semerci, Remi Galvez, Marcus Tannenberg, Sahitya Mantravadi, Divya Narayanan, Ofeliya Kalaydzhyan, Douglas Cole, Ben Carterette, Ann Clifton, Paul N. Bennett, Clau- dia Hauff, and Mounia Lalmas. Podtile: Facilitating podcast episode browsing with auto-generated chapters. ACM International Conference on Information and Knowledge Management (CIKM), 2024.
Google Deepmind. the agentic era.
Introducing gemini 2.0: our new ai model for
https:/ /bblog google/technologee wwttte/hnlogygggolll----e- google-demmmnnd/ google-gemini-a -i -p-aattte----------e -el---uddtt--dmeee--2222-2 december-2224 2024. Accessed: 2025-03-28.
Michael Gygli, Yale Song, and Liangliang Cao. Video2gif: Automatic generation of animated gifs from video. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Seungju Han, Jack Hessel, Nouha Dziri, Yejin Choi, and Youngjae Yu. Champagne: Learn- ing real-world conversation from large-scale web videos. In International Conference on Computer Vision (ICCV), 2023.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022.
Rosie Jones, Ben Carteree, Ann Clion, Maria Eskevich, G. Jones, Jussi Karlgren, Aasish Pappu, Sravana Reddy, and Yongze Yu. Trec 2020 podcasts track overview, In Text Retrieval Conference (TREC), 2020.
10

---

Published as a conference paper at COLM 2025
Jussi Karlgren, Rosie Jones, Ben Carterette, Ann Clifton, Edgar Tanaka, Maria Eskevich, G. Jones, and Sravana Reddy. Trec 2021 podcasts track overview. In Text Retrieval Conference (TREC), 2021.
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Yu Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. Dense passage retrieval for open-domain question an- swering. In Conference on Empirical Melhods in Natural Language Processing (EMNLP), 2020.
Minsun Kim, Dawon Lee, and Junyong Noh. Generating highlight videos of user-specified length using most replayed data. In CHI Conference on Human Faciors in Computing Systems (CHI), 2025.
Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D. Plumbley. Panns: Large-scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2880-2894, 2019.
Chia-Hsuan Lee, Shang-Ming Wang, Huan-Cheng Chang, and Hung yi Lee. Odsqa: Open- domain spoken question answering dataset. In IEEE Spoken Language Technology Workshop (SLT), 2018.
Chyi-Jiunn Lin, Guan-Ting Lin, Yung-Sung Chuang, Wei-Lun Wu, Shang-Wen Wen Li, Abdelrah- man Mohamed, Hung yi Lee, and Lin shan Lee. Speechdpr: End-to-end spoken passage retrieval for open-domain spoken question answering. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024.
Kevin Lin, Pengchuan Zhang, Joya Chen, Shraman Pramanick, Difei Gao, Alex Wang, Rui Yan, and Mike Zheng Shou. Univtg: Towards unified video-language temporal grounding. In International Conference on Computer Vision (ICCV), 2023.
Ye Liu, Siyuan Li, Yang Wu, Chang Wen Chen, Ying Shan, and Xiaohu Qie. Umt: Unified multi-modal transformers for joint video moment retrieval and highlight detection. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.
Meta. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models.
https://all /aa::///iiia meta. commblog/llamaaa 3- - (oannect--22---- 202---222ii--- wwweeiiseieeeeniiieeeeeee mobile- devices/, 2024. Accessed: 2025-03-07.
Marco De Nadai, Francesco Fabbri, Paul Gigioli, Alice Wang, Ang Li, Fabrizio Silvestri, Laura Kim, Shawn Lin, Vladan Radosavljevic, Sandeep Ghael, David Nyhan, Hugues Bouchard, Mounia Lalmas, and Andreas Damianou. Personalized audiobook recommen- dations at spotify through graph neural networks. In ACM on Web Conference (WWW), 2024.
OpenAL. Hello gpt-4o. http::///poonaaaccccc////e/lells 4pth://hhhlllgot-oo.ooom 2024a. Accessed: 2025- 02-05.
OpenAL. Gpt-4o mini: advancing cost-efficient intelligence. https://opentt wttppppenn..cm/innn/eeeeeee gpt-4o-minis -ini-advancing-----. cost
wtt.:/oo//cii// com/index/ 2024b. Accessed: 2025-03-01.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Tlya Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning (ICML), 2022.
Stephen F. Robertson and Hugo Zaragoza. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends Information Retrieval, 3(4):333--33)) 2009.
James A Russell and Albert Mehrabian. Evidence for three-factor theory of emotions. Journal of Research in Personality, 11(3):2732294 1977.
Yale Song, Jordi Vallmitiana, Amanda Stent, and Alejandro Jaimes. Tvsum: Summarizing
web videos using titles. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
11

---

Published as a conference paper at COLM 2025
Jinhwan Sul, Jihoon Han, and Joonseok Lee. Mr. hisum: A large-scale dataset for video high- light detection and summarization. In Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL htttps://api... www.nti org/CorpusID: 268042559.
Hao Sun, Mingyao Zhou, Wenjing Chen, and Wei Xie. Tr-detr: Task-reciprocal transformer for joint moment retrieval and highlight detection. In AAAI Conference on Artificial Intelligence (AAAI), 2024.
Min Sun, Ali Farhadi, and Steven M. Seitz. Ranking domain-specific highlights by analyzing edited videos. In European Conference Computer Vision (ECCV), 2014.
Mi Tian, Claudia Hauff, and Praveen Chandar. On the challenges of podcast search at spotify. In ACM International Conference on Information and Knowledge Management (CIKM), 2022.
Johannes Wagner, Andreas Iriantafyllopoulos Hagen Wierstorf, Maximilian Schmitt, Felix Burkhardt, Florian Eyben, and Björn W Schuller. Dawn of the transformer era in speech emotion recognition: closing the valence gap. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 45(9):10745-10755 2023.
Bohong Wu, Zhuosheng Zhang, Jinyuan Wang, and Hai Zhao. Sentence-aware contrastive learning for open-domain passage retrieval. In Annual Meeting of the Association for Computational Linguistics (ACL), 2021.
Bo Xiong, Yannis Kalantidis, Deepti Ghadiyaram, and Kristen Grauman. Less is more: Learning highlight detection from video duration. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Antoine Yang, Arsha Nagrani, Ivan Laptev, Josef Sivic, and Cordelia Schmid. Vidchapters- 7m: Video chapters at scale. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Jin Yang, Ping Wei, Huan Li, and Ziyang Ren. Task-driven exploration: Decoupling and inter-task feedback for joint moment retrieval and highlight detection. In iEE.//CV Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Nan Yang, Furu Wei, Binxing Jiao, Daxin Jiang, and Linjun Yang. xmoco: Cross momentum contrastive learning for open-domain question answering. In Annual Meeting of the Association for Compulational Linguistics (ACI.), 2021. URL https: //api. semanticscholar. org/CorpusID:2230 233455996.
YouTube. Youtube video player updates: Most replayed, video chapters, single loop & more. 2025-02-25.
google.
2022. Accessed:
Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jizee Cao, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Mingyao Zhou, Wenjing Chen, Hao Sun, and Wei Xie. Cross-modal multiscale difference- aware network for joint moment retrieval and highlight detection. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024.
12

---

Published as a conference paper at COLM 2025
A
Data Collection Details
A.1 Data Filtering
Filtering Channel IDs From the list of channel metadata gathered from the popular podcast creators webpage on YouTube, we exclude channels that satisfy at least one of the following conditions.
Default language is set and is not English.
"defaul tL tLanguage" " in channel_mo md[ "snippet"] and channel_md "mmmeel_emmssiiptett]l
tLaggugge" != "en"
In this step, we keep channels that do not have default language set and they are further filtered in the following step in playlist-level their content is not in English.
Among all publicly available playlists of the channels from the previous step, we only keep playlists that satisfy all of the following conditions:
Filtering Playlist IDs
Creators have explicitly set them as podcasts. "podcastStatus" in playlist_mdil statusss and playIist_mmd""tatus""""
podcastStatus") == "enabled"
Does not contain any filter words its title. a[wo[w not in (ngliistmmm"sniippet"[tt" reaction", "review", "breakdown"]])
for word in ["music",
The playlist assumed be in English.
isEnglish(' ([aaaaaalsasmmmmmsiiieeeeeeeeeee snippet"[[" "hhnn""cccaallileeeeee
playlist_md[" powplist_mmd"sniippp" "I"" "["'description"]]]
To filter non-English playlists out, we first concatenate the playlist title, channel title, and playlist description, using separator as space- identifier string. Next, we utilize Linguaf, lightweight library for language identification, to analyze the identifier string. Playlists are assumed to be in English (1) the highest-confident language is English or (2) has confidence score lower than 75%. The second condition added based on the authors' manual inspection of filtering results to avoid false negatives.
hnttps://github http://thhhhcccmmmmmsssaalllllll
13

---

Published as a conference paper at COLM 2025
A.2
Converting Replay Graph into Set of Highlight Indices
K 20, 0.0
K = 20, 0.5
1.00
0.50
0.00
0.75
0.25
0.75
0.25
1.00
0.50
0.00
0.75
mulammn
0:5
1.00
0.50
0.00
1.00
0.75
0.50
0.25
0.00
0.75
0.25
1.00
0.50
0.00
0.75
0.55
Figure 7: More examples comparing two different hyperparameter settings for converting replay graph into of GT highlights.
14

---

Published as a conference paper at COLM 2025
A.3
Bias Correction in Replay Graphs
1.0
Corrected
0.10
Replay Store
0.6
0.4
0.06
Highlight
0.2
0.02
0.0
0.00
0
20
40
80
100
20
40
60
80
100
Segment Index
Segment Index
Figure 8: (Left) The average replay graph over all replay graphs in RHAPSODY. The shaded area represents the standard deviation at each index. (Right) The highlight selection rate for each segment index before and after bias correction.
As shown in the left graph in Figure 8, the replay graphs internally have biases, particularly at the first segment. Specifically, the first value (0.711) is significantly higher than the rest, which ranges from 0.054 to 0.208. This pattern likely arises because users frequently replay the beginning of the content after missing the start, rather than due to the segment's inherent saliency. Since this spike reflects an artifact of the user's watching behavior rather than meaningful engagement, we correct it by setting the first value equal the second value, iee graph[o] = graph[1]. As shown in the right plot of Figure 8, this correction has little effect on the distribution of highlights in segments other than the first. The average number of highlights per episode GG] decreases slightly from about 5.86 to 5.20.
B Input Data Preprocessing Details
B.1 Transcript Segment Summarization
Algorithm describes our multi-stage summarization pipeline, which aims to improve the quality of summaries and coherence across transcript segment summaries. Given 100 transcript segments, the algorithm processes them in batches of 10. At each stage, the LLM generates individual summaries for the batch and a combined summary incorporating contextual information from the previous batch. The first stage has no prior context, so it starts with an empty summary.
Algorithm Multi-Stage Summarization
1: Input: Transcript segments {4]] 100
2: Initialize: So +- "None.", (f,) uninitialized array of size 100
3: for = 1 to 10 do
10(k-1)+10 4: B₁ + [1,) 110
5: S + LLM(M,, (;
6: Store B into corresponding indices in {f11,00
7: end for
8: Output: Summarized transcript segments [1)] summaries for each batch [8x]10
The full prompt used can be found in Appendix C.1.
15

---

Published as a conference paper at COLM 2025
B.2
Audio Feature Extraction
Valence
Dominance
Arousal
Nor-highlight Highlight
Non-tighliihht
Non-highlight Highlight
Densiyy
Highlight
0-
0.00
0.25
0.50
0.75
1.00
0.0
0.4
0.6
0.8
0.25
0.00
0.50
0.75
1.00
Figure 9: Valence, dominance, arousal score density distributions for highlight and non- highlight segments. Dotted vertical lines represent the mean of the distribution.
We use otttr/////iiiieeeeeeeeerrooororr tion, and Facebok//uberttbbas---sg66 for HuBERT feature extraction. The model produces 1024-dimensional and 512-dimensional embedding from input audio file, respectively, To obtain 100 embeddings per episode, we first load the raw audio at a 16,000Hz sample rate, split it into 100 segments, and process each through the model.
for DVA feature extrac-
Specifically, the DVA embeddings can be further used to predict valence, dominance, and arousal scores with their pretrained regression head. As shown in Figure 9, there is slight but obvious difference in all dimensions. For example, the average score is slightly higher for highlighted segments across all dimensions.
C Used Prompts
We denote the variables in the template with [teletype font and curly brackets).
C.1 Multi-Stage Summarization of Transcript Segments
SYSTEM You expert in summarizing podcast transcripts into concise summaries.
Your tasks are:
1. Summarize each provided segment, ensuring coherent logical flow.
2. Generate a combined summary for all the provided segments, focusing on conveying the
overall context efficiently without unnecessary detail.
3. If previous summary is provided, ensure your summaries align with it.
Focus on providing a concise overview, omitting minor details when necessary to ensure clarity.
Produce output in the following format:
Segment X: [Summaar of Segment X] Segment [Summary of Segment Y]
Segment Z: [Summary of Segment Z]
Combined Summary: [Summary of all segments combined]
USER Summarize the following podcast transcript into individual segment summaries and combined summary for all the provided segments.
Podcast Title: (title)
Previous Summary (if any): (combined summary of the previous batch}
Segments:
Segment 0: (transcript of segment 1}
transcript of segment to in the same format)
Segment 9: (transcript of segment 10)
Table 5: Prompt template for summarizing transcript segments.
16

---

Published as a conference paper at COLM 2025
C.2 Zero-Shot Prompting with GPT-4o and Gemini
SYSTEM You are an assistant trained to analyze summaries of podcast segments and predict listener behavior. Your task is to identify segments that are most likely to be replayed, rank them by likelihood, and provide concise and specific rationales. Follow these guidelines when answering:
1. Rank segments in descending order of replay likelihood.
Include a segment only if there is highly obvious reason it might be replayed, excluding uncertain segments.
If no segments meet the criteria, leave the Answer' item empty.
4. Respond in the following format:
Segment (one-line rationale for why replayed)
Segment Y: (one-line rationale for why it is replayed)
Segment Z: (one-line rationale for why it is replayed)
Answer: X, Y,.., Z
USER Analyze the following podcast segments:
Title: {title}
Segment 0: (summary of transcript segment 1}
Segment 1: (summary of transcript segment 2}
(summary of transcript segment 3 to 98 in the same format)
Segment 98: {surmary of transcript segment 99}
Segment 99: (summary of transcript segment 100)
Which segments are most likely to be replayed?
Table 6: Text-only prompt template for identifying highlight segments.
SYSTEM You an assistant trained to analyze summaries of podcast segments and predict listener behavior. You will be provided the summaries of podcast segments as well as the original podcast audio. Your task is identify segments that are most likely be replayed, rank them by likelihood, and provide concise and specific rationales. You should use both the provided summary text and the audio to make your decision, paying attention to the content of the text and audio well as prosodic and emotional cues from the audio. Follow these guidelines when answering:
1. Rank segments in descending order of replay likelihood.
Include a segment only if there is highly obvious reason it might be replayed, excluding uncertain segments.
3. If no segments meet the criteria, leave the Annwwer' item empty.
4. Respond in the following format:
Segment X: (one-line rationale for why it is replayed)
Segment Y: (one-line rationale for why is replayed)
Segment (one-line rationale for why it replayed)
Answer: X, Y, Z.,, Z
USER (podcast audio} Analyze the following podcast segments:
Title: [title}
Segment 0: (summary of transcript segment 1}
Segment 1: (summary of transcript segment 2}
(summary of transcript segment to 98 in the same format)
Segmeet 98: (surmary of transcript segment 99}
Segment 99: (summary of transcript segment 100)
Which segments are most likely to be replayed?
Table 7: Text audio prompt template for identifying highlight segments.
17

---

Published as a conference paper at COLM 2025
D Model Implementation Details
Zero-shot prompting with LLMs For both got-4o-2024-08-08 and gemini-2. 0-llash-00,, we use default hyperparameter settings. The string follow- ing Answer: in the LLM response is parsed and converted into a set of predicted indices.
Fine-Tuned LLM with Token-Selective Classification Head The model is trained using batch size of and a gradient accumulation step of 8. QLoRA is applied with a dropout rate of 0.3, rank of and alpha value of 16. The optimizer used AdamW with weight decay of 0.01 and learning rate of 0.0003. We train the model with NVIDIA RTX8000 GPUs. The model outputs the indices the graph above a certain threshold as predictions. We use the prediction threshold of 0.3, 0.425, and 0.375 for text-only, text + DVA, and text + HuBERT settings, respectively. These values are determined by the authors' manual tuning on the validation set. We take checkpoints with the best average metrics on the validation set.
E Attribution of Icons Used
All the icons used in this paper are from https://www.flattee
Location
Icon
Author
Wave Sound
Freepik
Document
Freepik
Check
hqrloveq
Figure 1
Delete
hqrloveq
Code
pikepicture
Robot
Hilmy Abiyyu A.
Freezing
Freepik
Figure 3
Fire
Bahu Icons
Table 8: Author information for all icons used.
18


Reviews:
[{"review_id": "yhigSDwHiI", "reviewer": "Reviewer_csU1", "summary": "This paper introduces a new dataset of 13K podcasts paired with which segments are highlights for training and evaluating models on highlight prediction. The evaluation covers both LLMs and trained multimodal and unimodal models, showing that current zero-shot LLMs perform relatively poorly but trained models can recover some signal, though there is still substantial room for improvement. Combing multiple modalities also seems to help.\n\nThe paper is highly original and introduces a challenging task that benefits from multiple modalities. The evaluation is thorough and represents a solid amount of work for a COLM paper. The dataset will likely be used by later tasks and has nice parallels with video highlight prediction; however, given the lack of work in audio for podcasts, this resource will likely help inspire much future work in the area. The paper is clearly written and well executed.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- New challenging task that benefits from text and audio modalities\n\n- New resource for the community\n\n- Thorough evaluation that covers multimodal and unimodal LLMs, both off the shelf and a custom model that is fine-tuned", "reasons_to_reject": "The paper is very good so I don't have any strong reasons to reject. However, there are a few weaknesses/critiques in the comments box.", "questions_to_authors": "- The only notable weakness for the paper for me is a lack of an error analysis, which might help guide how to improve performance. \n  - Podcasts can vary significantly in length so I wonder if there are issues with model performance relative to how long the segments are. With a mean length of ~30 minutes, does the model perform worse on 60 minute podcasts?\n  - The authors have a nice note about the issue of the first segment frequently being a highlight. I would be curious if there is additional structure in the podcast itself that might point to varying error rates by segment number\n\n- I would have liked to see if the authors could get a bit more performance from the commercial LLMs. I still think the fine-tuned model is likely going to be better, but I do think tweaking the prompts some could help. For example, the authors note on Line  208 that the zero shot models could not be controlled for how many highlights they create. However, I could imagine a prompt that informs the model about what the median number of highlights are, or suggests a range of highlights. Prompt engineering can be a rabbit hole, so I'm not suggesting that the authors endlessly tweak their design. However, I do think some stronger prompting designs might tell us a bit more about where the gap is in their performances and what aspects need to be improved.\n\n- Since Figures 4 and 5 are pooling episodes with same number of highlights, I would appreciate seeing errors bars on the plots \n\n- Line 256 notes the 100K podcast dataset of Clifton et al. (2020) as the largest but I think this has been superseded by the 1.1M podcasts dataset of [Litterer et al. (2024)](https://arxiv.org/abs/2411.07892) which is on huggingface https://huggingface.co/datasets/blitt/SPoRC", "extra_scores": {}}, {"review_id": "M1G7t9yakG", "reviewer": "Reviewer_7EoE", "summary": "This paper tackles the challenge of highlight detection in podcasts using podcast transcripts, audio features, and episode titles. The authors create a new dataset that maps YouTube videos to replay graphs that highlight the most watched portions of podcasts. They evaluate models like GPT4o, Gemini and, LLama on this longform comprehension task.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-- This paper works on an interesting and important problem.", "reasons_to_reject": "-- The proposed bias correction mechanism could over-correct, i.e., cases where the first frame is indeed a highlight may not be correctly recognized due to the de-biasing employed. \n\n-- How accurate are the V/A/D scores, the transcripts and the VAD scores ? Errors in these processes could artificially inflate the complexity of this task. \n\n-- The current formulation of the task involves multiple automatic steps to predict possibly derived replay graphs. It is possible that with multimodal LLMs like Gemini used in this paper to do this task in a more direct and simple manner. For example, one could use a Gemini model to transcribe only the highlights with timestamps or transcribe the entire audio but add special tags around highlighted segments etc. It is not clear that the proposed formulation is the best one.", "questions_to_authors": "-- Did the authors consider using visual information as well ? Could some peaks in the replay graph be correlated with events in the visual scene that may be ignored by the current approach ?\n\n-- Did the authors try this task with audio only ? Models like Gemini are capable of predicting timestamps to the best of my knowledge, and those could be used to correlate directly with the replay graph. \n\n-- What happens if peaks occur at the boundary of segments ?\n\n-- Why not use the transcript segments in full rather than their summaries ? \n\n-- It is not clear if authors have the legal permissions to re-distribute the most replayed feature graphs for public consumption. Specific licensing information should be included in the paper.", "extra_scores": {}}, {"review_id": "scRygb3uQi", "reviewer": "Reviewer_ZtNz", "summary": "This paper explores the challenge of automatically identifying highlights in podcasts. It introduces RHAPSODY, a dataset of 13,000 podcast episodes. The authors frame highlight detection as a binary classification task and use a set of models including zero-shot prompting and fine-tuned LLMs for this task. Experimental results show that even advanced models like GPT-4o and Gemini do not perform well in highlight detection.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1- The introduction of the RHAPSODY dataset is a valuable contribution. There are a few podcast datasets available in the community and this dataset can extend the research in this very important field.   \n\n2- The task of highlight detection is also new and well-motivated, providing a benchmark that the community can build upon.\n\n3- By combining speech signal features and transcripts, the paper provides evidence that multimodal representations significantly improve performance. This insight could inform future research in multimodal and spoken content understanding.", "reasons_to_reject": "1. While highlight detection in video content often aligns with visually or emotionally engaging moments, podcast highlights are inherently semantic and contextual. They typically reflect importance, novelty, or narrative turning points, I believe these are qualities that are better captured by extractive summarization approaches like TextRank. This distinction is not explored in the paper, which can be an interesting addition. Additionally, different podcast formats (e.g., interviews, solo speaker, storytelling) may require different definitions and strategies for highlight detection. The authors should acknowledge this variation and discuss its implications for model training and evaluation.\n2. Podcasts span a wide range of genres, each with a unique structure and style. These differences can influence highlight characteristics. However, the paper does not evaluate how genre affects model performance, which limits our understanding of the model’s generalizability and robustness.\n3. Many podcasts are published with creator-written descriptions or summaries, which are used in other benchmark datasets (e.g., Spotify) as ground-truth references [1]. The paper does not specify whether these summaries were collected, and/or does not justify the decision to disregard these human-authored annotations.\n4. Podcasts often contain ads, sponsor messages, intros, or other repetitive boilerplate sections that are not semantically meaningful [2]. The paper does not explain whether such segments were removed or how the models are expected to handle them. These segments could add noise to both the training and model predictions.\n5. The reported performance is relatively low, even for large models like GPT-4o and Gemini, which raises concerns about the task formulation and the reliability of the evaluation method. While the finetuned models perform better, they still struggle with precision. This may indicate that the models are not capturing what human listeners actually perceive as highlights. Since the evaluation is completely automatic and based on replay-based supervision, it remains unclear whether replay peaks truly align with human judgment. Incorporating human evaluation of predicted highlights would strengthen the paper and help validate the task formulation and model outputs.\n\n\n[1] Ann Clifton, Sravana Reddy, Yongze Yu, Aasish Pappu, Rezvaneh Rezapour, Hamed Bonab, Maria Eskevich, Gareth Jones, Jussi Karlgren, Ben Carterette, and Rosie Jones. (2020). 100,000 Podcasts: A Spoken English Document Corpus. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5903–5917, Barcelona, Spain (Online). International Committee on Computational Linguistics.\n\n[2] Reddy, S., Yu, Y., Pappu, A., Sivaraman, A., Rezapour, R., & Jones, R. (2021, April). Detecting Extraneous Content in Podcasts. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume (pp. 1166-1173).", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mDA04QTsLx", "reviewer": "Reviewer_HBck", "summary": "This paper introduces RHAPSODY, a dataset of 13K podcast episodes for highlight detection, where \"highlights\" are derived from YTB's \"most replayed\" feature. The authors formulate highlight detection as a segment-level binary classification task and benchmark various commonly used approaches.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper introduces the first large-scale dataset for podcast highlight detection; The use of YouTube's replay graph as a proxy for highlights and does not require manual labeling.\n2. The authors evaluate a well-designed range of approaches from simple heuristics to retrieval methods (BM25) to state-of-the-art LLMs (GPT-4o, Gemini) in both zero-shot and fine-tuned settings. This provides a thorough baseline for future works.", "reasons_to_reject": "1. While the paper demonstrates that audio features improve performance, it lacks sufficient justifications: only one type of audio embedding is explored (VDA emotion embeddings from wav2vec2, Section 3.1 and Appendix B.2), without justification for this specific choice. No ablation studies on alternative audio representations (e.g., raw spectrograms, CLAP embeddings, or other pre-trained audio models)\n\n2. The paper's writing could be improved: (1) The contribution list (lines 53-60) should follow a more logical order: (a) problem formulation, (b) dataset creation, (c) benchmark systems; (2) The motivation for the task is underdeveloped in the introduction; (3) using \"Feature-agnostic\" (line 108) to categorize different methods is misleading, (4) System descriptions are scattered across Sections 3.1 and 3.2 (are data preprocessed in section 3.1 used in all of the systems in sec3.2? if not, that step is not data preprocessing but one steo of the system); (5) The distinction between \"comparison systems\" and experimental setup is unclear, from what I saw, the baselines are just random/frequency based methods, and the rest are all proposed systems. \n\n3. The evaluation metrics lack clear definitions and explanations; The discussions on \"Analysis on the Number of GT and Predicted Highlights\" (lines 232-247) and \"Performance Across Varying Number of GT Highlights\" (lines 248-252) are particularly confusing: (1) Figure 4's purpose and interpretation are unclear - why is it important that models predict similar numbers regardless of |G|? (2) The transition between discussing quantity (Table 3, Figure 4) and performance metrics (Figure 5) lacks coherence. These analyses feel disconnected from the main narrative about model performance", "questions_to_authors": "1. Why were emotion embeddings used for audio features? What motivated this choice over other audio representations?\n2. Can you provide qualitative examples of segments that are incorrectly classified as highlights to better understand model behavior?\n3. How do results vary across different podcast genres?\n4. What is the human performance on this task?", "extra_scores": {}}]

Author rebuttal:

Meta-review: Summary of the work:
The work introduces the Rhapsody dataset, consisting of 13k audio podcasts based on YouTube podcast videos. The authors propose to use the "most replayed" metadata provided with Youtube videos to assign "highlight segments" automatically, which is proposed as a challenging task (as the authors argue highlights can help listeners navigate podcasts efficiently). Experiments conducted show poor performance of out-of-the-box methods (ZS LLMs, etc.) and that fine-tuning (including models that leverage both audio and text modality) leads to improvements, though the task remains challenging.

Recommendation:
All reviewers lean positive, and I tend to agree. The idea of using "most replayed" signal, and the successful implementation at scale, is a solid contribution to the community. I recommend the authors carefully integrate discussion points in their draft, including:
- Analysis around podcast length vs. performance and genre
- Additional discussion points, including around human annotation.

Additionally, I strongly recommend that the authors carefully consider how to release the dataset thoughtfully. As mentioned by one of the reviewers, it could be unclear whether "most replayed" metadata is allowed to be scraped and redistributed by Google. Also, it might be thoughtful to allow podcast creators to "opt out" from being included in the collection; this option was available for the Spotify podcast dataset (for example).
Decision: Accept