Bilingual conversation (chinese, english):

这篇文章提到了翻译里边的formality的问题. Formality的定义是什么呢?如何理解它呢? So we consider different labels for different formality levels. It's essentially a classification problem where... Well, it's not a classification problem in translation, but there are different classes of formality, and since English itself doesn't have different formality levels, or at least explicitly... Whatever. Yeah. So that means basically different level of formalness, let's say, how you want to have your translation. So if the question is the language coverage, then there are two stages. One is the pre-training, one is the task-specific fine-tuning on formality-controlled translation. In pre-training, we're not changing it, so it's just a no language left behind or NLLB 200 model out of the box. 啊,这个意思啊. 然后我看在这篇文章里你涉及到了多种的语言, 然后, 这是在我理解的情况下, 这是一个非常实际的问题, 尤其是在不同的语言的情况下. 我的接下来的问题是, 有哪些语言在这篇文章里涉及到然后被实验过呢? when you translate into different languages with different formality levels, then this translation is somewhat underspecified, right? So this means we kind of have to control whether if you go from English to German, whether we are going to formal German or informal German. Like, you know, in German, there's the formal you versus the informal you, and then the verbs coming along are also differently inflected. Is that the word? Inflection? As the name says, it covers 200 languages. And I don't have the list, but it covers high and low resource languages. And then in fine tuning, I believe for formality, we had like five target languages like English to five target languages in general and then the test was done on different languages as a zero-shot condition. I think for the zero shots, languages, as in the languages that do not have formality labels, then these would be, if I remember correctly, Russian, Portuguese, and Korean. Then we also have experiments on gender, grammatical gender control, and these I think there we were a bit restricted by the data set, so we only had like higher resource languages like Spanish and French. And maybe something else that I forgot. But then the setup is that we train the control on some language and then try to test the performance on these zero-shot or artificial zero-shot languages. 在这篇文章它并不是从随机初始的模型来开始训练的, 而是在你刚才说的NLLB的模型上开始继续训练的. 这样做的目的是什么呢? Because, I mean, without a pre-trained model that can already do just general translation for those languages, it probably would... then we are having a bigger problem of zero-shot just learning zero-shot translation and zero-shot attribute control translation. I don't think it'll work. And also basically the question we ask in the paper is, if you have an existing pre-trained model that can do general translation on many directions, if you then additionally learn, like on the figure one, if you additionally learn say attribute control translation on only a small set of languages, to what extent can you transfer this to the other different languages that are covered by the model? 你刚才提到了如果必须要从预训链开始. 然后在我的理解下就是, 如果你有足够多的数据的话, 即使是从随便初始的模型来说的话, 它也会有比较不错的效果. 所以说我猜这篇文章是不是涉及到了一些小数据量的一些数据集. 是这个原因吗? I think the question has two parts. The first part is whether you can just train a massively multilingual model like NLLB with, say, your own data and own compute, right? I don't think it's realistic and I also don't think it's practical. I mean, of course you can say, okay, now if we only consider those 10 languages and informality control, then we do like a, what is it, we train a model from scratch and do a 10 to 10 language multilingual system and then do control. It is also possible, but the performance, I mean, In one of the experiments, we actually had one system trained from scratch just to basically ablate out of the effect of having massive multilingual pre-training. So you can still do it. I think it's somewhere later on in like table four or something. So there we compare just NLLB versus like I think 10 languages or something, normal transformer trained from scratch. So the kind of transfer techniques still work, but I think in general they aren't as performant as using NLLB out of the box. So I mean, of course, it's a bigger model, NLLB, and it's also distilled from a much even bigger model. But yeah, I think in... basically your question has two layers. One is just the data amount for general translation, and the other is the data amount for, say, attribute annotated translation, as in how much data you have for a certain language where you know certain target languages are, say, formal and certain are informal. I mean, since it's very specific, I would say in general this kind of, attribute specific data is low resource compared to general translation data. Does that answer the question partly? 是的. 啊那我明白了这样做的原因. 而且你有一个很好的对比是，是否用预训练和不用预训练. 然后我觉得这个 好的. 然后我接下来的另一个问题是, 我看到你的模型的翻译的方向涉及到了从英语到多个语言之间. 因为语言与语言之间的差异很大, 然后它们的formality之间的差异也会很大, 我好奇你有没有发现不同的语言的翻译的方向会有一些不同的结果? 有一些不同的发现和不同的结论? Yes, you are right. I mean, different languages, every language is different. And even there are, you know, cultural connections as to what formality really is, right? For example, But if you consider some lower-resourced ones, there are still quite some problems. So with that in mind, I kind of think that it's still relevant to pursue this kind of direction with more conventional encoder-decoder models without LLMs. I read that in Korean, formality is not a binary thing, like informal versus formal, but instead there are several layers or levels. So like, like basically with close friends, you do this with maybe your colleagues, you do this and with, I don't know, bosses or, or seniors, you do something else. And I mean, here I think we only have zero, we mainly have zero shot results on Korean. No, we actually also have the oracle results by using a bit of supervised data, but in general, as you can see, if we look at the control scores, that is the average accuracy next to the column of blue scores. They aren't, even if it's supervised, this is not to be compared with the numbers above, but the table two or three. 在刚才的回答里你提到了Zero-Shot, 那么在这篇文章里Zero-Shot的定义是什么呢? 它是这个模型先在一些数据集里进行训练, 然后再用到另一个数据集里直接来做推测吗? on the page before. Like if you have these languages like German and Italian then they are nearly perfect. So as for the definition of zero-shot, what is zero-shot is attribute specific translation for a certain translation direction. just general translation for that direction is not zero-shot. It's already covered in the pre-trained model. What we assume is that for, say for formality control, there is data, from a different language, different translation direction. We can learn basically the notion of what is formal, what is informal by learning on this data. And then at inference time, what, 恩, 是的 we are trying to achieve is to do the same kind of control on a different translation direction that doesn't have the kind of supervised attribute specific translation data. Is that 感谢. 然后我的下一个问题是, 我看到在衡量模型表现的时候你用到了两个衡量它的那个分数. 但是这两个分数, 如果我理解对的话, 它是在机器翻译里面常用到的. 这两个分数你觉得会很好的来表现formality control这个问题吗? Okay. 是的 You mean we are using BLEU and COMET to measure the translation quality? okay uh so actually there's in general a bit of a trade-off between control and quality. Um what we said earlier BLEU and COMET, these are only the like quality metrics But for control, we are actually measuring them by these accuracy scores. So what happens under the hood is that the dataset creators, they created these kind of annotations on certain say, formality or gender specific words. And basically we were just lexically matching those to see how well we are doing. So if you want to consider the performance of the final systems, maybe the most comprehensive way is to plot a curve and then basically to see how you can do trade-offs between 对的, 确实翻译的质量和它的准确性都很重要的, 在这个问题. 好的 感谢. 然后我还有一个问题就是, 现在的大语言模型它们的能力已经变得越来越强了, 甚至它们之后可能会变得更强. 然后你有没有想思考过大语言模型在这个 问题上他的表现会如何? 他是否能解决这个问题呢? 如果你给予一些比较好的提前的问题的引导 these like formality control accuracy or whatever metric otherwise measures it versus translation quality as measured by BLEU or COMET. Yeah, that's a very relevant question. I mean, one of the reviewers also actually asked about it. Basically, why don't you LLM it? Yes, it's definitely a very good use case, I would say, to... say, prompt a large language model to say, BLOOM, please generate me the translation from English to German in formal. I mean, I think Binh even tested this on Vietnamese and it somehow works well. But again, one question, as we said earlier, is the trade-off between quality and control. So, at least at the time of writing LLMs for translation, there was still quite a bit of performance gap between dedicated systems like NLLB and just using LLMs. I think there was a paper from WMT last year that basically found when translating into higher resource languages, chatGPTs and co. are doing well.