Bilingual conversation (english, german):

Okay, so let's talk about the paper Discrete SLU, a large language model with self-supervised discrete speech units for spoken language understanding. Should we start with summarizing the paper a bit? Ja, ich kann es zusammenfassen, wenn du willst. Also in dem Paper geht es um Discrete Speech Units und die haben halt geschaut, ob das verbessert oder nicht und sind dann am Ende auch zu dem Entschluss gekommen, dass es was hilft und ja, haben es auch mit anderen vorherigen Wissenschaften verglichen. And what is the general setting? Like what's the task they're trying to improve? Wait. The LLM token embedding space. Yeah, I guess the task they want to do is automatic speech recognition. So they want to improve the task of when somebody records speech and you write it down. And I think what people usually do is that they they record the speech and then they convert the recorded speech to like a continuous embedding space and what they do differently is that they don't use a continuous embedding space but instead they use a discrete speech unit Have you looked a bit into how these things get generated? Nein. Okay, so is there something else you want to talk about maybe? Vielleicht können wir darüber reden, wie es anders ist als anderes also als andere, die davor gemacht wurden. That sounds good. Also wenn ich es richtig verstanden habe, dann war AudioPalLM, ich bin mir nicht sicher, wie man es ausspricht, das was davor verwendet wurde, aber da ist es anders mit Speech-Adapter und so umgegangen, wenn ich es richtig verstanden habe, also die haben es anders verwendet. Mhm Yeah, maybe you can tell me which section you're talking about right now Es ist bei Related Work der letzte Abschnitt for Proposed Approach, Punkt Nummer 3. Ah, okay. Yeah, I see. Yeah, I guess they also use discrete audio tokens, so it's kind of similar what they they have been doing. I was not so sure whether it's actually such a new approach because I think they used the HuBERT as an encoder. Is that right? Ich glaube schon, ja. Yeah, and I mean there has already been some work about HuBERT doing discrete audio tokens and I think at the HuBERT work they also use K-means which they do to get the discrete tokens. So I don't see there's a big difference to what they're doing. But you were earlier talking about the results, that they got better. Wenn ich mich richtig erinnere, dann waren sie besser, aber manchmal auch nicht. Also das war das, was ich nicht ganz verstanden habe. Also manchmal hat es was geholfen, aber manchmal auch nicht. Ich glaube das ist halt einfach noch nicht ausgearbeitet genug und sie müssen noch ein bisschen daran arbeiten. Okay. Yeah, I didn't look at the tables in detail, to be honest. But yeah, I was more interested in the method itself. So I think what they also do is they write something about deduplication. Deduplication involves converting sub-sequences of consecutive repeated class sequences to a single unique index. So maybe that's interesting. But I'm not really sure. And then about the method how they trained the models, have you read something about that? Ähm... Weißt du, wo das steht? I think it's in section 3.2 like in the proposed approach Das habe ich ehrlich gesagt übersprungen, weil es Spanisch für mich war oder nicht so genau gelesen. Yeah, it's maybe technical. So, is there something you'd like to talk about then? Vielleicht kannst du es kurz zusammenfassen, wenn du willst. Okay, so I think what they do is they somehow convert this audio into the recording. And then they use this recording and convert it to embeddings that are continuous and then they cut it to discrete ones. Okay, so let's talk a bit about instruction tuning of the speech adapter, section 3.2. That's where we left off the last time. Is that okay for you? Ja. Ah ja, auf jeden Fall. Okay, so I think what they do is... So the input to the model is usually a prompt like: 'Can you transcribe this audio for me?' Yeah. and then the audio. So the text portion of the prompt is just tokenized into subwords, like in smaller. parts of the word and then it gets embedded with the LLM's embedding layer. And then the same happens for the audio, apart from it's not the LLM embedding there, but they convert it to continuous embeddings with some other encoder, I assume you would, but it doesn't say here. Yeah, and then they just concatenate it together and feed it into the model. And then they For the audio part, there's also some adapter between the speech tokens and the LLM. And they train the adapter between the speech tokens and the LLM, and then they also put a low rank adaptation, LoRA, to the LLM. Vielleicht können wir uns die Ergebnisse anschauen. Mhm, yeah, sounds good. So what metrics do they use to evaluate the task? Ähm. WER und Bleu Mhm. Do you know what WER stands for? Or what it measures? A word embedding. Und dann noch. Word Error Rate. So I think it measures how many errors the model makes when transcribing the audio. Also man sieht ja, dass das erste, das ist J2, also dieses Wave LM21 am besten ist. Bei WER und bei Bleu ist dann ja das zweite am besten. Ich finde, es macht eben keinen Sinn, dass der unterschiedlich am besten ist. Und die anderen waren ja generell nicht so gut. Doch das letzte war ganz unten noch gut. Ich weiß nicht, machen die Ergebnisse Sinn für dich? I'm not sure, why do they use BLEU anyways? Maybe they also have, like usually use BLEU for translation tasks? So maybe they also do speech translation and not only ASR? Maybe that's what they... maybe they use a different test set there? I'm not sure. Oh maybe, oh no, I think that's spoken language understanding that they do there. So it's basically you hear the audio and then you ask the model I don't know, what did you hear in the audio? Can you answer the following question? Also Chain of Thought? So bisschen? No, I think in a chain of thought it's more you ask the model a question that it can answer to help with the next question. But in this it's more like, imagine you have an audio about, I don't know, the weather. And then the model would hear the audio and then you would ask the model, what was the weather yesterday? And the model is supposed to know because it heard it in the audio. Ah, okay, ja. I would not word by word transcribe the whole thing, but just would answer to the question. Okay, ja. And in Chain of Thought, maybe you can tell me what you did with Chain of Thought. So. Äm, also ich habe, um dem Modell zu helfen, weil es nicht so gute Ergebnisse hatte, habe ich gefragt, dass es mir erst, also das Modell musste das Geschlecht erkennen und dann habe ich erst gefragt, welches Geschlecht siehst du? Und mit diesem Wissen sollte es dann den Satz übersetzen, damit es halt eben Hilfe hat und damit ich verstehe, wo es das Geschlecht gut erkennt und wo es das Geschlecht nicht gut erkennt. Yeah. Ja. Okay, I guess in this setting maybe you could come up with something like that. Like you could ask, can you transcribe the part where the speech talks about some specific topic and then after transcribing it, you can ask a question towards it. But I think that's not what they did here. So I think it's not super surprising that different models have different strengths, so depending how they are trained, one could be really good at doing question answering and then The other one could be really good in just writing down what it heard. Mhm, ja, das macht auch Sinn. Yeah. Gehört das, weißt du, ob das MFCC auch zu denen gehört? to the models. Ja, weil das ja ganz anders heißt. Yeah, this is an L model. What was L again? Let me see. Yeah, so I think, yeah, it doesn't say it. I can't find it right now. Okay, you got the second part?