Please, Translate the following English into Korean: 
Hello everybody, my name is <PRESIDIO_ANONYMIZED_PERSON>. I'm a 4th year PhD candidate at MIT, advised by Professor Song Kang. Today it's my great pleasure to introduce AWQ, Activation of Way-Away Contposition for Large-Range Model Compression and Acceleration. So large-range models are very powerful, however they are very expensive as well. So these models have been widely deployed in today's everyday applications such as chatbots and software development. However, you usually notice that there are some messages from the websites of OpenAI and what is demonstrated in this figure. So basically the largest GPU available now is the NVIDIA B200 GPU. So it has only 192GB of memory. However, the largest transformer model that has been publicly announced by Jensen Huang in GTC 2024 actually has 1.8 trillion parameters. So it is necessary for us to use some techniques to reduce the model sizes and make sure they are deployable on real devices. So naturally quantization can help reduce the deployment costs because if you want to serve a 3.7 billion model, it actually requires 140GB of memory if you just directly serve this model in FP16. However, if you are able to quantize this model to INT8 in terms of the weights or INT4, you are able to significantly reduce the deployment cost from like two A100 GPUs with 80GB of memory all the way down to one A100 GPU with 40GB of memory which is actually also much cheaper. So previously our group has done some pioneering work in terms of the quantization of large-range models which is called SmoothQuant. So the idea of SmoothQuant is that the large-range models, their quantization difficulty actually is very different for the weights and the activations. So specifically the activations are very hard to quantize due to the number of outliers presented. So very lucky we can actually transfer the quantization difficulty from activations to weights through an operation called Smoothing which is multiply the weights by a scaling factor that is larger than one and then multiply the inverse of this scaling factor to the activations. So W888 quantization is definitely good. It reduces the cost for the memory required and improves the peak throughput of large-range model serving since the models can be mapped onto INT8 and so forth with like 2X higher throughput. However, is W888 quantization enough for large-range model deployment especially on the edge? The answer is no. So basically in this figure we actually presented the root flag model for the RTX 49 GPU which is very common in the desktop AI applications. So basically the x-axis of this figure is the computation intensity of the workload. So here we use the general matrix multiplication as an example. So here the computation intensity in terms of max per element is equivalent to the batch size of the gem workload. So in this example we can clearly see that there is a demarcation point at batch size equal to roughly 80 for this specific device. So this means that if the batch size goes beyond 80, then W888 quantization is more beneficial because the peak throughput of the INT8 Tensor Core is much higher than that of the FP16 Tensor Core. However, if we go below this value, we are entering the memory bound region. So in this case, the peak throughput of the battery Tensor Core does not guarantee the battery peak performance of the entire workload. So from a more global point of view, we actually break down the large-range model inference into two stages. The first stage is called the context stage which is actually much faster than the generation stage or the autoregressive decoding stage. That is because the context stage can actually be run in one pass using very high parallelism and can be mapped onto the Tensor Cores for efficient computation. However, when it comes to the generation stage, as I have just mentioned, the process is actually autoregressive and tokens are generated in a one-by-one manner. So this means that our computation intensity is just one max per element. Basically, this lies in the sweep region of the weight-only quantization instead of the weight and activation quantization because the memory bandwidth is the more important factor for the final performance of the entire workload. In terms of the memory traffic, we actually also break down it into the weight traffic and activation traffic. Not very surprising because the large-range models are very large in terms of the weights and the batch size is very small so the weight traffic dominates the entire traffic. To put all these things together, to accelerate large-range models on the edge for the batch size equal to one inference, it is very important for us to reduce the weight traffic so it becomes very natural for us to design algorithms that perform weight-only quantization. Let's jump into AWQ. We start our experiments from a very simple variant that is run-to-nearest quantization. We found that if we quantize the large-range models directly using the monotonearest method to very low bits like 3 bits, we actually observe very significant accuracy loss. Specifically, the perplexity of the OPT7B model actually degrades from 10ish to 40ish. This basically means that the entire model is not usable after this quantization. It's natural for us to think about strategies to solve this problem. Basically, we found that if we unquantize some of the channels from this entire weight tensor by identifying some of the salient channels, we are able to well-preserve the quantization accuracy. As you can see from this figure, if we keep 1% of the salient channels unquantized, the perplexity actually goes back from 40ish to 10ish again. A natural question is how do we select these salient channels? Should we select these channels based on weight magnitude, activation magnitude, or shall we just randomly select these channels? It turns out that the selection of the salient channels has a very significant impact on the accuracy of the large-range model quantization. Basically, we found that very surprisingly, as it's guided by the conventional waste-to-meeting and weight-pruning community, if you want to quantize the waste, it's definitely important to look at the weight distributions like these large magnitude weights. However, our experiments show that if you select the salient channels based on the magnitude of waste, there is actually no improvement over random selection. However, if we go from weight-based selection to activation-based selection, that means you select these channels based on the largest activations, you can see that actually the performance goes pretty significantly towards the value direction. Basically, the key takeaway from this slide is that in order to identify important channels in large-range models, we should actually look at the activation distribution instead of the weight distribution. Another follow-up question is, is it really necessary for us to design mixed precision methods for both computation and system? Because actually mixed precision inevitably will introduce some complexity in terms of both algorithms and systems. For example, you have to design real-world reading and you have to split the kernels into two parts, one for the FP16 computation and one for the integer multiplier with the floating point. The answer is fortunately yes, we do have a way to avoid mixed precision. Basically, here we actually developed a method called protecting the salient weights by scaling. The idea is actually quite simple. After we identify the salient channels by the activation magnitude, we are actually able to scale up these salient channels. Please notice it's scaling up instead of scaling down. If we scale up these channels by 1.5x, you can see that actually the accuracy is going back quite quickly. If we continue to increase the scale two times, the perplexity actually continues to improve. If we have a too large scale factor for these salient channels, the perplexity goes worse again. This is a very interesting observation and we definitely want to understand why. To understand this phenomenon, why AWQ is able to reduce quantization errors through scaling up these salient channels, we have to start from the basic definition of quantization. In quantization, we first need to calculate a term delta called the quantization scalar. This term is defined by the absolute maximum value of the entire weight tensor divided by the number of quantization levels. After we have this value, we divide all the weights by this value and then run it to integer. We apply the scales again to convert the continuous values to the discrete values. With that, we are able to express each term of the partial sounds in the matrix of a function in the original models in the following form. We can see that if we apply these manufacturers, the entire form of the equation is very similar to the original form. Except the fact that the original exhibition was scaled down by a factor of S and S is greater than 1. We can do some error analysis on these equations. For these three terms, for both deltas, the scalar and the X, the original activations, they are all represented in floating point, so no errors for these terms. Error only comes from the rounding term. We have an analysis on the distribution of the error term and we found out that the quantization scalar delta did not change after we scaled up the salient channels. The highlighted intuition behind that is that if you only scale up some of the channels within the entire weight tensor, this will not change the value in the entire tensor. So, there is no change. For the error term of the rounding term, we found that this is actually a scalar with expectation of 0.5. No matter what is the value inside this rounding term, the expectation is always 0.5. The product of the first two terms is actually a scalar. Basically, the error term is totally determined by the third term, which is X versus X divided by S. There is an assumption for this analysis, which is actually S is not too large. That is because if you have a very large S, you can imagine that even if you are just scaling up some of the channels, the absolute max of the entire tensor will scale up. Basically, we have to have a reasonable value of S. This necessitates our scaling search in AWQ. Basically, very interestingly, our scaling search is also guided by the difference in the activation instead of the difference in the weights. Actually, I think this also shows the importance of activation awareness in terms of determining the optimal scaling factor. We do find that such decision based on the activation difference is better than the decision based on the weight difference. AWQ is very accurate, very simple, easy to implement, has good hardware efficiency, and it has less dependency on the calibration dataset. We will show later that the method is also applicable to visual language models, even if we only calibrate the model on the text-made corpus. AWQ achieves really good results in terms of general large-length model quantization. It achieves very good results compared with the mountaineers quantization and also very strong baselines like GPT-Q with reordering. We also show that AWQ is applicable for visual language models. These visual language models actually convert visual images into tokens through VITs. These visual tokens are concatenated with the text-to-tokens and processed by the large-length models join. Basically, the visual language models are very powerful. I think most of you have watched the GPT-4.0 debut last day. They definitely show some video understand demos. These applications require very strong performance of the large-length models and very good efficiency because you want to interact with these models in real time. Our method is going to quantize the visual language models also to focus on the weights and significantly improve their execution efficiency. As demonstrated in this table, we actually well-preserve the accuracy of visual language models after quantization. For some of the benchmarks such as MMBat, we even observe some performance improvements after the quantization. After we develop AWQ, we definitely want to mention that it is a non-trivial task for us to deploy these large-length models on the edge. Here comes the system part. We designed a very lightweight large-length model inference engine called TinyChat, which is a Pythonic, lightweight, efficient, and most importantly, supports multiple platforms. We want to take the best of both worlds from the community. For example, we want to be as easy to use as the Hugging Face transformers while at the same time deliver the great efficiency provided by TensorIT OM from NVIDIA Corporation. Basically, we have two very important findings. The first one is that to design an efficient inference engine for different hardware platforms, it is very important to design a weight-packing format based on the characteristics of the platform. For example, we have different packing formats for Intel CPUs, ARM CPUs, and NVIDIA GPUs. The major reason is that for different platforms, they have different intrinsics. For example, we take our Intel CPUs as an example. We are able to reduce the number of operations that is required to decode 32 weights all the way down to 3 logical operations through our weight reordering or repacking. This brings about 1.2 times the speedup on the CPU and has similar effects on the NVIDIA GPUs with TensorFlow, especially for the context-stage with Jam implementations. We also have extensively implemented kernel fusion for our GPU backend. Two very important implementations to mention is that we have used the tension for all the workloads. We fused everything in the tension operator into a single kernel, which we found to be very useful on the desktop GPUs. We also have fused Jam and Jamd kernels. We actually used Jam kernels for the context-stage and Jamd kernels for the decoding stage and the dequantization process from H4 weights to FP16 weights are fully fused into these two kernels. TinyChat significantly accelerates the edge large-range model deployment. It offers more than 3 times the natural speedup over the hugging phase FP16 large-range model inference. At the same time, we are as flexible as hugging phase transformers. We are at least 2.6 times faster than MotoGPQ and it is more flexible than llama.cpp and exLlama, which the generalizing names are specialized for the llama models. TinyChat is a lightweight large-range model inference engine and can be deployed almost everywhere. We actually worked with students from the Power School of Design to manufacture this little box with a GPU board with just 7GB of available memory. Our system is able to deploy 7GB of large-range models in this device. This is definitely very important because we want to value data privacy and security. We want to have continued operation of large-range models in disconnected environments and we do want to support customization and personalization by deploying these models on the engine. We can also see this as four personal laptops with both Intel and ARM CPUs. As you can see in this example, we are able to deploy the large-range models and LASM counters to write sorting programs in C. We do have very significant speedups on different GPU platforms, especially on the linear adjustment offering. We achieved more than 3 times speedup over the FP16 implementation and it provides a really interactive VLM deployment experience as is demonstrated in this video. We are trying to deploy the VLM 7GB model on this ORI device and we asked the model what is unusual in this image. The model is able to identify that there is a group of data across the world and we continue to ask whether we should proceed. The model is able to reason about the image and say that it is not a good idea to proceed. Basically, we are also able to deploy the visual language models on the CPU backends. This is a Macbook with ARM CPU. We are able to interact with the model with the word knowledge, for example, what is the name of the painting and what is the author of the painting. AWQ has generally a significant community impact. We have been downloading for over a million times on Hugging Face. We are deployed in NVIDIA's TensorISER OM, Hugging Face Transformer, and we are the default quantization backend for the IBM Red Light model which was just released last week. We are also integrated into LMBeployment, VLM, and Google Vertex AI. UC Berkeley has also integrated our system into the FastShop, our LMS system. We are also integrated into the Intel-Ural-PC. In summary, AWQ accelerates large-angle deployments through algorithm and system co-design. We discovered that low-bit wave quantization improves the efficiency of large-angle models recorded on the edge. We found that scaling up the salient channels, which is determined by deactivation magnitude, is very important for us to reduce the large-angle quantization error. We also find that, especially for GPU platforms, fused curls improve the efficiency for memory-bound workloads such as large-angle models on the edge. For the next step of AWQ, we have also recently developed a new method called Qserve, which is the W4A8KE4 quantization method for the cloud serving of the large-angle models. This system is very efficient and outperforms the NVIDIA TensorID LLM on both A100GPU and L4ESGPU. We are able to match the performance of NVIDIA TensorID LLM on 3X cheaper L4ES with our backend. With that, I am going to conclude my talk today. Thanks everyone for attending my talk. Questions are welcome. Thank you.