Please, Translate the following English into Korean: 
Hello everybody, my name is <PRESIDIO_ANONYMIZED_PERSON>. I'm a fourth year PhD candidate at MIT, advised by Professor Song Han. Today it's my great pleasure to introduce AWQ, Activation-Away-Away Composition for Large-Range Model Compression and Acceleration. So large-range models are very powerful, however they are very expensive as well. So these models have been widely deployed in today's everyday applications such as chatbots and software development. However, you usually notice that there are some messages from the websites of OpenAI and Anthropic saying that currently we are having very high demand and we cannot provide the service as you expected. So definitely one of the largest challenges in terms of deploying the large-range models is that they have really large sizes. So despite being very powerful, large-range models have outgrown the pace of the hardware capacity growth, especially like what is demonstrated in this figure. So basically the largest model is the one that is the largest in the world. The largest GPU available now is the Nvidia B200 GPU. So it has only 192GB of memory. However, the largest transformer model that has been publicly announced by Jensen Huang in GTC 2024 actually has 1.8 trillion parameters. So it is necessary for us to use some techniques to reduce the model sizes and make sure they are deployable on real devices. So naturally quantization can help reduce the deployment costs. Because if you want to serve a LAMA3, a CES, or a T-Series model, you need to have a lot of data. So if you want to serve a LAMA3, a 70 billion model, it actually requires 140GB of memory if you just directly serve this model in FP16. However, if you are able to quantize this model to INT8 in terms of the weights or INT4, you are able to significantly reduce the deployment costs from like 2 A100 GPUs with 80GB of memory all the way down to 1 A100 GPU with 40GB of memory, which is actually also much cheaper. So previously, our group has done some pioneering work. In terms of the quantization of large range models, which is called SmoothCount. So the idea of SmoothCount is that the large range models, their quantization difficulty actually is very different for the weights and activations. So specifically, the activations are very hard to quantize due to the number of outliers presented. So we are very lucky we can actually transfer the quantization difficulty from activation to weights through an operation called smoothing, which is multiplying the weights by scaling factors. So we can actually scale the factor that is larger than 1 and then multiply the inverse of this scaling factor to the activations. So W8A8 quantization is definitely good. It reduces the cost for the memory required and improves the peak throughput of large range model serving, since the models can be mapped onto INT8 and so forth with like 2x higher throughput. However, is W8A8 quantization enough for large range model deployment, especially on the edge? The answer is no. So basically, in this figure, we actually presented the roofline model for the RTX 4090 GPU, which is very common in the desktop AI applications. So basically, the x-axis of this figure is the computation intensity of the workload. So here we use the general matrix multiplication as an example. So here the computation intensity in terms of max per element is equivalent to the batch size of the general workload. So in this example, we can clearly see that there is a demarcation point at batch size equal to roughly 80 for this specific device. So this means that if the batch size goes beyond 80, then W8A8 quantization is more beneficial because the peak throughput of the INT8 tensor core is much higher than that of the FP16 tensor core. However, if we go below this value, we are entering the memory bound region. So in this case, the peak throughput of the, you know, the bare tensor cores actually does not. So this means that the entire workload is guaranteed by the peak performance of the entire workload. So from a more global point of view, we actually break down the large-range model inference into two stages. The first stage is called the contact stage, which is actually much faster than the generation stage or the autoregressive decoding stage. That is because the contact stage can actually be run in one pass using very high power of the man can be mapped onto the tensor cores for efficient computation. However, when it comes to the generation stage, as I have just mentioned, the process is actually autoregressive and tokens are generated in a one by one manner. So this means that our computation intensity is just one max per element. So basically, this lies in the sweet region of the weight only quantization instead of the weight and activation quantization, because the memory bandwidth is the more important factor for the final performance of the entire workload. So in terms of the memory traffic, we actually also break down it into the weight traffic and activation traffic. And not very surprising because the large-range models are very large in terms of their weights, and the batch size is very small, so the weight traffic dominates the entire traffic. So put all these things together to accelerate large-range models on the edge for the batch size equal to one inference, it is very important for us to reduce the weight traffic, so it becomes very natural for us to design algorithms that performs weight only quantization. So let's jump into AWQ. So we start our experiments actually from a very small batch size. So we have a very simple variant, that is, the Ron-Tunier quantization. So we found that if we quantize the large-range models directly using Ron-Tunier's method, to very low bits like 3 bits, we actually observe very significant accuracy loss. Specifically, the perplexity of the OPT7B model actually degrades from 10-ish to 40-ish. So this basically means that the entire model is not usable after this quantization. So it's natural for us to think about strategies to solve this problem. So basically we found that if we unquantize some of the channels from this entire weight tensor, by identifying some of the salient channels, we are able to well preserve the quantization accuracy. So as you can see from this figure, if we keep 1% of the salient channels unquantized, the perplexity actually goes back from 40-ish to 10-ish again. So a natural question is how do we select these salient channels? Do we select these channels based on weight magnitude, activation magnitude, or shall we just randomly select these channels? So it turns out that the selection of the salient channels has a very significant impact on the accuracy of the large-range model quantization. So basically we found that, very surprisingly, as it's guided by the conventional wisdom in the weight pruning community, actually if you want to prune the weights or you want to quantize the weights, it's definitely important to look at the weight distribution, select these largest magnitude weights. However, our experiments show that actually if you select the salient channels based on the magnitude of weights, there is actually no improvement over random selection. However, if we go from weight-based selection to activation-based selection, that means you select these channels based on the largest activations, you can see that actually the performance goes pretty significantly towards the better direction. So basically the key takeaway from this slide is that in order to identify important channels in large-range models, we should actually look at the activation distribution instead of the weight distribution. So another follow-up question is, is it really necessary for us to design mixed precision methods for both quantization and system? Because actually mixed precision inevitably will introduce some complexity in terms of both algorithms and systems. For example, you have to design reordering and you have to split the kernels into two parts, one for the FP16 computation and one for the integer multiplied by weight. So what is the answer to this question? So the answer is fortunately yes, we do have a way to avoid mixed precision. So here, we actually developed a method called protecting the salient weights by scaling. So the idea is actually quite simple. After we identify the salient channels by the activation magnitude, we are actually able to scale up these salient channels. Please notice it's scaling up instead of scaling down. So if we scale up these channels by 1.5x, you can see that actually the accuracy is going back quite quickly. And if we continue to increase the scale to two times, the perplexity actually continues to improve. But if we have a too large scale factor for these salient channels, actually the perplexity goes worse again. So this is a very interesting observation and we definitely want to understand why. So basically, to understand this phenomenon, why AWQ is able to reduce quantization errors through scaling up these salient channels, we have to start from the basic definition of quantization. So in quantization, we first need to calculate a term, delta, called the quantization scalar. So this term is defined by the absolute maximum value of the entire weight tensor divided by the number of quantization levels. So after we have this value, we actually divide all the weights by this value and then round it into integers. And then we apply the scales again to convert the continuous values to the discrete values. So basically, with that, we are able to explain the problem. And we can express each term of the partial sums in the matrix multiplication in the original models in the following form. So basically, we can see that if we apply the spin factors, the entire form of the equation is actually very similar to the original form. Except the fact that the original activation was scaled down by a factor of s, and s is greater than 1. So we can actually do some error analysis on these equations. Basically, for these three terms. Actually, for both delta, the scalar, and the x, the original activation, they are all represented in floating point. So no errors for these terms. So error only coming from the rounding term. So basically, we have an analysis on the distribution of the error term. And we actually found out that the quantization scalar delta actually did not change after we scaled up the salient channels. The high level of integration behind that is that if you only scale up some of the channels within the entire weight tensor, this will not change the variance value in the entire tensor. So delta does not change. And for the error term of the rounding term, we found that this is actually a scalar with expectation of 0.5. So no matter what is the value inside this rounding term, the expectation is always 0.5. So the product of the first two terms is actually a scalar. So basically, the error term is totally determined by the third term, which is x versus x divided by s. So there is a scalar. So there is an assumption for this analysis, which is actually s is not too large. That is because if you have a very large s, you can imagine that even if you just scale up some of the channels, the absolute max of the entire tensor will scale up. So basically, we have to have a reasonable value of s. So this necessitates our scaling search in AWQ. So basically, very interesting, our scaling search is also guided by the difference in the activation. Instead of the difference in the weights. So actually, I think this also shows the importance of activation awareness in terms of determining the optimal scaling factor. We do find that such decision based on the activation difference is better than the decision based on the weight difference. So AWQ is very accurate, very simple, easy to implement, has good hardware efficiency, and it has less dependency on calibration data set. And we will show later that the method is also accurate. So we can see that it is very applicable to visual language models, even if we only calibrate the model on the text-weight corpus. So AWQ achieved really good results in terms of general large-range model composition. It achieves very good results compared with the mountaineers composition and also very strong baselines like GPTQ with reordering. We also show that AWQ is applicable for visual language models. So these visual language models actually convert the visual images into tokens through VITs. And then these visual tokens are concatenated with the text-weight tokens and processed by the large-range models joining. So basically, the visual language models are very powerful. I think most of you have watched the GPT-40 debut last day. And they definitely show some video understanding demos. So these applications actually require very strong performance of the large-range models and very good efficiency because you want to interact with these models in real time. So our method is going to be very similar to the one we used in the previous video. So our method is going to be very similar to the one we used in the previous video. We are going to quantize the visual language models also to focus for the weights and significantly improve their execution efficiency. So as is demonstrated in this table, we actually well preserve the accuracy of visual language models after quantization. And for some of the benchmarks such as MMBET, we even observe some performance improvements after the quantization. So after we develop AWQ, we definitely want to mention that it is a non-trivial task for us to deploy these large-range models on the market. So after we develop AWQ, we definitely want to mention that it is a non-trivial task for us to deploy these large-range models on the market. So after we develop AWQ, we definitely want to mention that it is a non-trivial task for us to deploy these large-range models on the market. So here comes the system part. We designed a very lightweight large-range model inference engine called TinyChat, which is PySonic, lightweight, efficient, and most importantly, supports multiple platforms. So we want to take the best of both worlds from the community. For example, we want to be as easy to use as the Hugging Face transformers while at the same time delivering the great efficiency provided by Tensor IT LOM from Nvidia Corporation. So basically, we have two very important findings. So basically, we have two very important findings. The first one is that to design an efficient inference engine for different hardware platforms, it is very important to design a weight packing format based on the characteristics of the platform. For example, we have different packing formats for Intel CPUs, ARM CPUs, and Nvidia GPUs. And the major reason is that for different platforms, they have different intrinsics. For example, we take ARM NEO CPUs as an example. We take ARM NEO CPUs as an example. We take ARM NEO CPUs as an example. And we can use them to reduce the number of operations that are required to decode 32 weights all the way down to 3 logical operations through our weight reordering or repacking. So this brings about 1.2 times the speed of the ARM CPU and has similar effect on the Nvidia GPUs with Tensor cores, especially for the context stage with gem implementations. So we also have extensively implemented kernel fusion for our GPU backend. So two very important findings. One of the important implementations to mention is that we do have fused attention for all the workloads. We fuse actually everything in the attention operator into a single kernel, which we found to be very useful on the desktop GPUs. And we also have fused gem kernel and gem v kernels. So we actually use the gem kernels for the context stage and gem v kernels for the decoding stage. And the dequantization process from int4 weights to FP16 weights are fully fused into these two kernels. And TinyChat actually significantly improved the performance of the kernel. And TinyChat actually significantly improved the performance of the kernel. There So TinyChat is a lightweight large- repository inference engine and can be deployed almost everywhere. So TinyChat is a lightweight large- simulator inference engine and can be deployed almost everywhere. So we actually work with students from Harvard School of Design to manufacture this little box with a GPU board with just 7GB of available memory. And our system is able to deploy 7 billion parameter large-ranging models in this device. So this is definitely very important because we want to value data privacy and security. We want to have continued operation of large-ranging models in disconnected environments. And we do want to support customization and personalization by deploying these models on the engine. So we can also seamlessly support personal laptops with both Intel and ARM CPUs. As you can see in this example, we are able to deploy the large-ranging models and let them help us to write sorting programs. So we do have very significant speedups on different GPU platforms, especially on the MN-Adjusted O-Ring. We achieved more than a 3x speedup over the FP16 implementation and provides a really interactive VLM deployment. As is demonstrated in this video. So we are trying to deploy the VLog7B model on this O-Ring device. And we ask the model what is unusual in this image. And the model is able to identify that there is a group of deer crossing the road. And we continue to ask whether we should proceed. And the model is able to reason about the image and say that, oh, it's not a good idea to proceed. So basically, we are also able to deploy the visual language models on the CPU back-end. So this is a MacBook with a VLog7B. And this is a MacBook with ARM CPU. So we are able to interact with the model with the word knowledge. For example, what is the name of the painting and what is the author of the painting. So AWQ has generated significant community impact. So we have been downloaded for over a million times on Hugging Face. And we are deployed in NVIDIA's Tensor IT LOM, Hugging Face Transformer. And we are the default compensation back-end for the IBM Core 9 model, which was just released last week. And we are also integrated into LLM Deploy. LLM and Google Vertex AI. And UC Berkeley has also integrated our system into the FastShot, our algorithm and system. So we are also integrated into the Intel and URO computers. So in summary, AWQ accelerates large-range deployments with algorithm and system co-design. So we discovered that low-bit weight quantization improves the efficiency of large-range model deployment on the edge. And we found that scaling up the salient channels, which is determined by the activation magnitude, is very important for us to reduce the large-range model quantization error. We also find that, especially for GPU platforms, fused kernels improves the efficiency for memory-bound workloads, such as large-range models on the edge. So for the next step of AWQ, we have also recently developed a new method called QSERV, which is the W4A8KE4 quantization method for the cloud serving of the large-range models. So this system is very efficient and all-performance in NVIDIA Tensor IT LOM. We can match the performance of NVIDIA Tensor IT LOM on both A100 GPU and L4ES GPU, and we are able to match the performance of NVIDIA Tensor IT LOM on the 3x cheaper L4ES with our backend. So with that, I'm going to conclude my talk today. Thanks everyone for attending my talk, and questions are welcome. Thank you.