Bilingual conversation (english, turkish):

Selam naber? I'm fine, thank you. What about you? Ben de iyiyim. Teşekkür ederim. Bu CVPR'da yeni çıkmış olan bir çalışma var, IPLAP. Biliyor musun? Yeah, I heard about it. Actually, I read it. Süper! Bunun hakkında biraz konuşabilir miyiz? Sana sormak istediğim bir şeyler var. Yes, yes, sure. Senin çalışma alanın olduğu için detaylı bildiğini biliyorum. açıkçası ben görüntüleri izledim videoları vesaire yaratılmış görüntüleri baya etkileyiciydi Mm hmm. Genel olarak kullandıkları yaklaşım öbürlerinden farklı. İki aşamalı bir yaklaşım kullanıyorlar sanırım, değil mi? Yes, they have two stage approaches. Actually, I also, I mean, they shared their models, weights and codes. So I also tested with my own models or our test models as you know. For our project and it looks like that their visual quality is pretty good. But I cannot say the same for lip synchronization. It is a bit not perfect at least not bad but maybe it is acceptable. But compared to other models, I think they are a bit worse, especially when you compare with the TalkLip or Wav2Lip. Yeah, I think... Peki ne düşünüyorsun? Yani ben baktığımda görsel kaliteyi kesinlikle çok iyi buldum. Yeah, the visual quality is pretty good. They don't have, I mean, the resolution is not high. They generate 128 by 128. so it is low quality of course but it is in the low quality domain they don't have any artifacts or other pose issues etc so their visual quality when you consider the resolution is pretty good But yeah, I mean generally there's a conflict between the visual quality and the lip synchronization. So if you I mean, for example, if you don't have very good lip synchronization, then it is relatively easier to have visual quality, but when you try to... Force model or push model to have better lip synchronization, then the visual quality issue happens. In this case, since they don't explicitly force the model to learn the synchronization very well, I think they don't have visual quality issues compared to other models. Anladım. İkisi arasında bir konflikt var aslında. Yeah, there's a trade-off, actually. Aynen, her zaman şey yapamıyoruz. İkisine de eşit oranda sahip olamıyoruz ve bunun üzerine bir ağırlık seçmiş oluyor. Yani bir tarafa yöneliyor. Yeah, at least right now the main research question is to have the very good visual quality, also perfect lip synchronization at the same time. Hatırladığım kadarıyla seninkisi daha mesela senkronizasyon üzerine odaklanıyor. Yeah, that's why we also have some visual quality issues compared to IPLAP. Anladım. Peki bu landmarkları kullanma, yüzdeki keypointleri kullanma fikrine ne Mm hmm. diyorsun? Genel olarak bazı çalışmalarda görüyorduk, yüz tanıma çalışmalarında. Burada da sanki etkili gibi. Yeah, I mean, as you know, it was very popular previously, but then they stopped using landmarks for this. But recently, I think one or recent in the last one or two years, it became popular again. I think it is kind of important for having very good visual quality, especially the pose alignment with the input video because normally you know that we have sometimes... The alignment issues, then you need to do post-processing, etc. But it might also cause some visual artifacts. But if you have the landmarks as a condition to the model, then it is relatively easier to have better visual quality. but i think in this paper they also use landmark prediction for lip synchronization in in the most most of the papers Diyorsun yani... Sadece landmarkları kullanarak mı ya da landmarklarla sesin ilişkisini ölçerek mi hesap yani... Yes, yes. ... senkronizasyonu sağlıyorlar? Yes, in most of the papers normally they use explicit synchronization loss to compare the audio and the generated visual data, mostly the lip area. to calculate whether there is a synchronization or not but in this paper they don't use it actually maybe that's another reason that they have very good visual quality because the current popular synchronization loss has some visual quality issues or causes some visual quality issues so they exclude that loss function and they try to predict the landmarks with the transformer model and based on the i mean they provide the audio as a condition so the network is responsible for predicting the correct lip landmarks based on the audio Ben de onu soracaktım. Hani... Loss fonksiyonlarında herhangi bir senkronizasyona özel bir loss fonksiyonu parçası var mı? İşte normalde bizim kullandığımız çalışmalarda ya da daha çalıştığımız çalışmalarda daha çok senkronizasyona özel bir optimizasyon da yapılıyor. Ama senin dediğin gibi sanırım burada daha çok direkt landmarkt. Yes, yeah. I mean there is no explicit synchronization loss. But the idea is to predict the correct landmarks, especially lip landmarks from the audio. And they compare the distance loss between the predicted landmarks and the ground truth landmarks. So implicitly they compare the distance between them. So they, okay, they kind of implicitly evaluate the synchronization or calculate the synchronization loss, but not directly and only through the landmarks, not the visual data. Anladım. Yani aslına bakarsan basitleştirmiş oluyorlar o açıdan da. Ama zaten... But I think that's why they have a synchronization issue. Evet, kesinlikle. Çünkü gördüğümüz kadarıyla önceki çalışmalardan her zaman özel olarak optimize etmek çok daha işe yarıyor. Ama dediğin gibi orada da öbür konflikti yaratıyor. Yani... Uh huh, uh huh. ... olan konflikti yaratıyor. Peki veri kümeleri vesaire, onlar standart veri kümeleri mi? Mesela eğitimi ne ile yapmışlar? I mean, they trained their model on LRS2 dataset, which is quite popular in talking face generation literature, I think. They tested in the third version of it as well, which is also relatively common. But I would expect HDTF dataset maybe or some other recently popular benchmark datasets because they also have high quality, high resolution data. So it is also good to test it. Visual capacity of the model, because these data sets are low quality, but at least they use two different data sets which are benchmark. Herhangi bir başka veri kümesi yok mu? Sanki bu görseller biraz HDTF'ten gibi duruyor ama değil herhalde. No, no, no, no. They are from the training data. Yeah, they said that. I mean I couldn't see any other data and there is no supplementary material in this paper but maybe they uploaded in the arxiv version, I don't know. Peki performansı ölçmek için yine benzer metrikler mi kullanmışlar? Şimdi bunlarda landmark Mm-hmm. Mm-hmm. Ölçümde belki farklı olarak vardı galiba. No, actually, they use the standard landmark distance metric for the mouth region. Normally, the other talking face generation models also use this metrics, although they don't use specific landmark generation or landmark loss. I think they used most of the benchmark metrics in the literature, mostly visual quality metrics. The PSNR, SSIM, FID, etc. For the synchronization I think they have this LMD, landmark distance loss, metric. It is not perfect to actually evaluate the synchronization. They also put under the reconstruction part actually. It is a bit complicated. It is a mix of... Lip synchronization evaluation and the visual quality evaluation, I think. But yeah, for the dubbing, they only have the synchronization score. Sadece lmd deme sebepleri, lip lmd deme sebepleri şey mi? Sadece o bölgeyi değerlendirdikleri için mi? Yani normal... Yeah. landmark Mm-hmm. ...dan farklı olarak. most in most of the papers in this literature, people also do like that. Because like, since you give the, I mean you mask the image generally, bottom half of it, because your model is responsible for generating the mouth region with respect to given audio. So when you mask bottom half, then you have the upper part. So it doesn't make too much sense to calculate the distance for entire landmarks. Because your model have the prior knowledge about the upper part. So it doesn't have any landmark issues there. Because of that, people focus on the mouth region only to calculate the lip sync or mostly focusing on the lip sync or at least If they calculate the visual quality part based on the landmark, still they want to focus on only the generated part. hı hı If you calculate for the entire phase, the score is getting very well. So I think it doesn't make sense. Kesinlikle bence de yani sadece ağız kısmına odaklanmaları çok mantıklı. Dediğin gibi bir de ayrıca senkronizasyon üzerinde yeah i think but there's only one metric they didn't Ama tablodan gördüğüm üzere en iyi sonucu alamamışlar zaten mantıklı. Çünkü... These scores are interesting actually because according to this table this table PC-AVS is the state of art in lip synchronization but Ama That is interesting. Evet, değil mi? Öbür çalışmalarda çok gördüğümüz bir şey değil. I think because of the metric, normally this confidence and distance metrics are more popular in the synchronization evolution. This sync score is a bit different than I. It is kind of similar with the confidence but not exactly the same. So when some papers use this metric But not recently anymore, actually. So I am surprised that in CVPR 2023, they published with only this metric. I don't know. Bilmiyorum kim değerlendirdi tabi de. Yeah. Aslında bir düşüneyim. Ha bir de şeyi sormak istiyordum son olarak. İkinci aşamada video rendering'de bu 3D'de vesaire kullanılan standart yaklaşımları mı kullanıyorsun? No, they do this 2D based, 2D based Evet 2D Base ama hani özel bir ya da ne bileyim daha önceden sunulmuş bir modeli mi kullanıyorlar diyeyim en azından hani. I mean I am not sure whether they exactly use one of the existing models but even if they design their own model it is still based on the existing State-of-the-art approach. They use generative adversarial network with some adaptive instance normalization layers and SPADE layers, which are quite popular in the recent papers in generative adversarial nets. Ama eğitmişler gibi duruyor. Yeah, of course, yeah. They just design model but they train with the bottom half masked input image as well as the predicted landmark map. They also have alignment module which they say that it's better. For visual quality generation, especially the consistency, temporal consistency for video generation. And in the alignment module, it's also kinda... actually... Bir de kalite yönünden mesela kaliteyi artırmak için bu standart işte Spade bloğu vesaire bu yapılar zaten. Dediğin gibi hani GAN based modellerde en çok kullanılan şeyler. Especially it is useful to combine this standard flow with the alignment module output. They inject the output of the alignment module. ...through the SPADE layer and for the audio through the adaptive instance normalization layer. Basit ama mantıklı bir şey Yeah, I think their main contribution is mostly landmark prediction part, not the rendering or the generation part. Doğru, evet, evet, kesinlikle. Okey, çok teşekkürler. Baya yardımcı oldu. You are welcome. Thank you for your questions.