Update README.md

README.md

@@ -4,18 +4,21 @@
 {}
 ---
 
-![](https://github.com/facebookresearch/encodec/raw/2d29d9353c2ff0ab1aeadc6a3d439854ee77da3e/architecture.png)
+![encodec image](https://github.com/facebookresearch/encodec/raw/2d29d9353c2ff0ab1aeadc6a3d439854ee77da3e/architecture.png)
 # Model Card for EnCodec
 
-This model card provides details and information about EnCodec, a state-of-the-art real-time audio codec developed by Facebook Research.
+This model card provides details and information about EnCodec, a state-of-the-art real-time audio codec developed by Meta AI.
 
 ## Model Details
 
 ### Model Description
 
-EnCodec is a high-fidelity audio codec leveraging neural networks. It introduces a streaming encoder-decoder architecture with quantized latent space, trained in an end-to-end fashion. The model simplifies and speeds up training using a single multiscale spectrogram adversary that efficiently reduces artifacts and produces high-quality samples. It also includes a novel loss balancer mechanism that stabilizes training by decoupling the choice of hyperparameters from the typical scale of the loss. Additionally, lightweight Transformer models are used to further compress the obtained representation while maintaining real-time performance.
+EnCodec is a high-fidelity audio codec leveraging neural networks. It introduces a streaming encoder-decoder architecture with quantized latent space, trained in an end-to-end fashion. 
+The model simplifies and speeds up training using a single multiscale spectrogram adversary that efficiently reduces artifacts and produces high-quality samples. 
+It also includes a novel loss balancer mechanism that stabilizes training by decoupling the choice of hyperparameters from the typical scale of the loss. 
+Additionally, lightweight Transformer models are used to further compress the obtained representation while maintaining real-time performance.
 
-- **Developed by:** Facebook Research
+- **Developed by:** Meta AI
 - **Model type:** Audio Codec
 
 ### Model Sources
@@ -30,22 +33,22 @@ EnCodec is a high-fidelity audio codec leveraging neural networks. It introduces
 
 <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
 
-EnCodec can be used directly as an audio codec for real-time compression and decompression of audio signals. It provides high-quality audio compression and efficient decoding.
+EnCodec can be used directly as an audio codec for real-time compression and decompression of audio signals. 
+It provides high-quality audio compression and efficient decoding. The model was trained on various bandwiths, which can be specified when encoding (compressing) and decoding (decompressing).
+Two different setup exist for EnCodec: 
+
+- Non-streamable: the input audio is split into chunks of 1 seconds, with an overlap of 10 ms, which are then encoded.
+- Streamable: weight normalizationis used on the convolution layers, and the input is not split into chunks but rather padded on the left.
 
 ### Downstream Use
 
-EnCodec can be fine-tuned for specific audio tasks or integrated into larger audio processing pipelines for applications such as speech recognition, audio streaming, or voice communication systems.
+EnCodec can be fine-tuned for specific audio tasks or integrated into larger audio processing pipelines for applications such as speech generation,
+music generation, or text to speech tasks.
 
 <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
 
 [More Information Needed]
 
-### Out-of-Scope Use
-
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
-
 ## How to Get Started with the Model
 
 Use the following code to get started with the EnCodec model:
@@ -69,13 +72,11 @@ reconstructed_audio = model.decode(audio_codes)
 
 ## Training Details
 
+The model was trained for 300 epochs, with one epoch being 2,000 updates with the Adam optimizer with a batch size of 64 examples of 1 second each, a learning rate of 3 · 10−4
+, β1 = 0.5, and β2 = 0.9. All the models are traind using 8 A100 GPUs.
+
 ### Training Data
 
-We train all models for 300 epochs, with one epoch being 2,000 updates with the Adam optimizer with a
-batch size of 64 examples of 1 second each, a learning rate of 3 · 10−4
-, β1 = 0.5, and β2 = 0.9. All the models
-are traind using 8 A100 GPUs. We use the balancer introduced in Section 3.4 with weights λt = 0.1, λf = 1,
-λg = 3, λfeat = 3 for the 24 kHz models. For the 48 kHz model, we use instead λg = 4, λfeat = 4.
 
 <!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
 
@@ -83,53 +84,45 @@ are traind using 8 A100 GPUs. We use the balancer introduced in Section 3.4 with
   - DNS Challenge 4
   - [Common Voice](https://huggingface.co/datasets/common_voice)
 - For general audio:
-  - AudioSet
-  - FSD50K
+  - [AudioSet](https://huggingface.co/datasets/Fhrozen/AudioSet2K22)
+  - [FSD50K](https://huggingface.co/datasets/Fhrozen/FSD50k)
 - For music:
-  - Jamendo dataset (Bogdanov et al., 2019)
-[More Information Needed]
+  - [Jamendo dataset](https://huggingface.co/datasets/rkstgr/mtg-jamendo)
+
 
-They used four different training strategies: 
+They used four different training strategies to sample for these datasets: 
 
-- (s1) we sample a single source from Jamendo with probability 0.32;
-- (s2) we sample a single source from the other datasets with the same probability;
-- (s3) we mix two sources from all datasets with a probability of 0.24;
-- (s4) we mix three sources from all datasets except music with a probability of 0.12.
+- (s1) sample a single source from Jamendo with probability 0.32;
+- (s2) sample a single source from the other datasets with the same probability;
+- (s3) mix two sources from all datasets with a probability of 0.24;
+- (s4) mix three sources from all datasets except music with a probability of 0.12.
 
-The audio is normalized by file and we apply a random gain between -10 and 6 dB. We reject any sample
-that has been clipped. Finally we add reverberation using room impulse responses provided by the DNS
-challenge with probability 0.2, and RT60 in the range [0.3, 1.3] except for the single-source music samples.
-For testing, we use four categories: clean speech from DNS alone, clean speech mixed with FSDK50K sample,
-Jamendo sample alone, proprietary music sample alone.
+The audio is normalized by file and a random gain between -10 and 6 dB id applied. 
 
 ## Evaluation
 
 <!-- This section describes the evaluation protocols and provides the results. -->
 
-We consider both subjective and objective evaluation metrics. For the subjective tests we follow the MUSHRA
-protocol (Series, 2014), using both a hidden reference and a low anchor. Annotators were recruited using a
+### Subjectif metric for restoration:
+
+This models was evalutated using the MUSHRA protocol (Series, 2014), using both a hidden reference and a low anchor. Annotators were recruited using a
 crowd-sourcing platform, in which they were asked to rate the perceptual quality of the provided samples in
-a range between 1 to 100. We randomly select 50 samples of 5 seconds from each category of the the test set
+a range between 1 to 100. They randomly select 50 samples of 5 seconds from each category of the the test set
 and force at least 10 annotations per samples. To filter noisy annotations and outliers we remove annotators
 who rate the reference recordings less then 90 in at least 20% of the cases, or rate the low-anchor recording
-above 80 more than 50% of the time. For objective metrics, we use ViSQOL (Hines et al., 2012; Chinen
-et al., 2020) 2
-, together with the Scale-Invariant Signal-to-Noise Ration (SI-SNR) (Luo & Mesgarani, 2019;
+above 80 more than 50% of the time. 
+
+### Objective metric for restoration:
+The ViSQOL()ink) metric was used together with the Scale-Invariant Signal-to-Noise Ration (SI-SNR) (Luo & Mesgarani, 2019;
 Nachmani et al., 2020; Chazan et al., 2021).
 
 ### Results
 
-We start with the results for EnCodec with a bandwidth in {1.5, 3, 6, 12} kbps and compare them to the
-baselines. Results for the streamable setup are reported in Figure 3 and a breakdown per category in Table 1.
-We additionally explored other quantizers such as Gumbel-Softmax and DiffQ (see details in Appendix A.2),
-however, we found in preliminary results that they provide similar or worse results, hence we do not report them.
-When considering the same bandwidth, EnCodec is superior to all evaluated baselines considering the
-MUSHRA score. Notice, EnCodec at 3kbps reaches better performance on average than Lyra-v2 using
-6kbps and Opus at 12kbps. When considering the additional language model over the codes, we can reduce
-the bandwidth by ∼ 25 − 40%. For instance, we can reduce the bandwidth of the 3 kpbs model to 1.9 kbps.
-We observe that for higher bandwidth, the compression ratio is lower, which could be explained by the small
-size of the Transformer model used, making hard to model all codebooks together.
+The results of the evaluation demonstrate the superiority of EnCodec compared to the baselines across different bandwidths (1.5, 3, 6, and 12 kbps). Figure 3 provides an overview of the streamable setup results, while Table 1 offers a category-wise breakdown. Although alternative quantizers such as Gumbel-Softmax and DiffQ were explored, their preliminary results did not surpass or match the performance of EnCodec, so they are not included in the report.
+
+When comparing EnCodec with the baselines at the same bandwidth, EnCodec consistently outperforms them in terms of MUSHRA score. Notably, EnCodec achieves better performance, on average, at 3 kbps compared to Lyra-v2 at 6 kbps and Opus at 12 kbps. Additionally, by incorporating the language model over the codes, it is possible to achieve a bandwidth reduction of approximately 25-40%. For example, the bandwidth of the 3 kbps model can be reduced to 1.9 kbps.
 
+Furthermore, it is observed that as the bandwidth increases, the compression ratio decreases. This behavior can be attributed to the small size of the Transformer model used, which makes it challenging to effectively model all codebooks together.
 
 #### Summary