README.md

---
license: apache-2.0
---

<div align="center">
  <img src="https://raw.githubusercontent.com/AXYZdong/AMchat/main/assets/logo.png" width="200" alt="AMchat Logo"/>
  <div align="center">
    <b><font size="5">AMchat</font></b>
  </div>
  <div align="center">
    <a href="https://github.com/AXYZdong/AMchat">💻Github Repo</a>
  </div>
</div>

## AMchat GGUF Model
AM (Advanced Mathematics) Chat is a large-scale language model that integrates mathematical knowledge, advanced mathematics problems, and their solutions. This model utilizes a dataset that combines Math and advanced mathematics problems with their analyses. It is based on the InternLM2-Math-7B model and has been fine-tuned with xtuner, specifically designed to solve advanced mathematics problems.


## Latest Release

2024-08-16

- **Q6_K**
- **Q5_K_M**
- **Q5_0**
- **Q4_0**
- **Q3_K_M**
- **Q2_K**


2024-08-09

- **F16 Quantization**: Achieves a balanced trade-off between model size and performance. Ideal for applications requiring precision with reduced resource consumption.
- **Q8_0 Quantization**: Offers a substantial reduction in model size while maintaining high accuracy, making it suitable for environments with stringent memory constraints.
- **Q4_K_M Quantization**: Provides the most compact model size with minimal impact on performance, perfect for deployment in resource-constrained settings.


## Getting Started - Ollama

To get started with AMchat in [Ollama](https://github.com/ollama/ollama), follow these steps:

1. **Clone the Repository**
   ```bash
   git lfs install
   git clone https://huggingface.co/axyzdong/AMchat-GGUF
   ```

2. **Creat Model**
   >Make sure you have installed ollama in advance. https://ollama.com/
   ```bash
   ollama create AMchat -f Modelfile
   ```
    
   
3. **Run**
   ```bash
   ollama run AMchat
   ```

## Getting Started - llama-cli

You can use `llama-cli` for conducting inference. For a detailed explanation of `llama-cli`, please refer to [this guide](https://github.com/ggerganov/llama.cpp/blob/master/examples/main/README.md)

### Installation

We recommend building `llama.cpp` from source. The following code snippet provides an example for the Linux CUDA platform. For instructions on other platforms, please refer to the [official guide](https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#build).

- Step 1: create a conda environment and install cmake

```shell
conda create --name AMchat python=3.10 -y
conda activate AMchat
pip install cmake
```

- Step 2: clone the source code and build the project 

```shell
git clone --depth=1 https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j
```

All the built targets can be found in the sub directory `build/bin`

In the following sections, we assume that the working directory is at the root directory of `llama.cpp`.

### Download models

You can download the appropriate model based on your requirements.
For instance, `AMchat-q8_0.gguf` can be downloaded as below：

```shell
pip install huggingface-hub
huggingface-cli download axyzdong/AMchat-GGUF AMchat-q8_0.gguf --local-dir . --local-dir-use-symlinks False
```

### chat example

```shell
build/bin/llama-cli \
    --model AMchat-fp16.gguf  \
    --predict 512 \
    --ctx-size 4096 \
    --gpu-layers 24 \
    --temp 0.8 \
    --top-p 0.8 \
    --top-k 50 \
    --seed 1024 \
    --color \
    --prompt "<|im_start|>system\nYou are an expert in advanced math and you can answer all kinds of advanced math problems.<|im_end|>\n" \
    --interactive \
    --multiline-input \
    --conversation \
    --verbose \
    --logdir workdir/logdir \
    --in-prefix "<|im_start|>user\n" \
    --in-suffix "<|im_end|>\n<|im_start|>assistant\n"
```

## Star Us
If you find AMchat useful, please ⭐ Star this repository and help others discover it!