TheBloke commited on
Commit
f07f4bf
1 Parent(s): 49e3373

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +0 -34
README.md CHANGED
@@ -59,7 +59,6 @@ Here is an incomplate list of clients and libraries that are known to support GG
59
  * [LM Studio](https://lmstudio.ai/), an easy-to-use and powerful local GUI for Windows and macOS (Silicon), with GPU acceleration.
60
  * [LoLLMS Web UI](https://github.com/ParisNeo/lollms-webui), a great web UI with many interesting and unique features, including a full model library for easy model selection.
61
  * [Faraday.dev](https://faraday.dev/), an attractive and easy to use character-based chat GUI for Windows and macOS (both Silicon and Intel), with GPU acceleration.
62
- * [ctransformers](https://github.com/marella/ctransformers), a Python library with GPU accel, LangChain support, and OpenAI-compatible AI server.
63
  * [llama-cpp-python](https://github.com/abetlen/llama-cpp-python), a Python library with GPU accel, LangChain support, and OpenAI-compatible API server.
64
  * [candle](https://github.com/huggingface/candle), a Rust ML framework with a focus on performance, including GPU support, and ease of use.
65
 
@@ -207,44 +206,11 @@ For other parameters and how to use them, please refer to [the llama.cpp documen
207
 
208
  Further instructions here: [text-generation-webui/docs/llama.cpp.md](https://github.com/oobabooga/text-generation-webui/blob/main/docs/llama.cpp.md).
209
 
210
- ## How to run from Python code
211
-
212
- You can use GGUF models from Python using the [llama-cpp-python](https://github.com/abetlen/llama-cpp-python) or [ctransformers](https://github.com/marella/ctransformers) libraries.
213
-
214
- ### How to load this model in Python code, using ctransformers
215
-
216
- #### First install the package
217
-
218
- Run one of the following commands, according to your system:
219
-
220
- ```shell
221
- # Base ctransformers with no GPU acceleration
222
- pip install ctransformers
223
- # Or with CUDA GPU acceleration
224
- pip install ctransformers[cuda]
225
- # Or with AMD ROCm GPU acceleration (Linux only)
226
- CT_HIPBLAS=1 pip install ctransformers --no-binary ctransformers
227
- # Or with Metal GPU acceleration for macOS systems only
228
- CT_METAL=1 pip install ctransformers --no-binary ctransformers
229
- ```
230
-
231
- #### Simple ctransformers example code
232
-
233
- ```python
234
- from ctransformers import AutoModelForCausalLM
235
-
236
- # Set gpu_layers to the number of layers to offload to GPU. Set to 0 if no GPU acceleration is available on your system.
237
- llm = AutoModelForCausalLM.from_pretrained("TheBloke/sqlcoder-GGUF", model_file="sqlcoder.Q4_K_M.gguf", model_type="starcoder", gpu_layers=50)
238
-
239
- print(llm("AI is going to"))
240
- ```
241
-
242
  ## How to use with LangChain
243
 
244
  Here are guides on using llama-cpp-python and ctransformers with LangChain:
245
 
246
  * [LangChain + llama-cpp-python](https://python.langchain.com/docs/integrations/llms/llamacpp)
247
- * [LangChain + ctransformers](https://python.langchain.com/docs/integrations/providers/ctransformers)
248
 
249
  <!-- README_GGUF.md-how-to-run end -->
250
 
 
59
  * [LM Studio](https://lmstudio.ai/), an easy-to-use and powerful local GUI for Windows and macOS (Silicon), with GPU acceleration.
60
  * [LoLLMS Web UI](https://github.com/ParisNeo/lollms-webui), a great web UI with many interesting and unique features, including a full model library for easy model selection.
61
  * [Faraday.dev](https://faraday.dev/), an attractive and easy to use character-based chat GUI for Windows and macOS (both Silicon and Intel), with GPU acceleration.
 
62
  * [llama-cpp-python](https://github.com/abetlen/llama-cpp-python), a Python library with GPU accel, LangChain support, and OpenAI-compatible API server.
63
  * [candle](https://github.com/huggingface/candle), a Rust ML framework with a focus on performance, including GPU support, and ease of use.
64
 
 
206
 
207
  Further instructions here: [text-generation-webui/docs/llama.cpp.md](https://github.com/oobabooga/text-generation-webui/blob/main/docs/llama.cpp.md).
208
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
209
  ## How to use with LangChain
210
 
211
  Here are guides on using llama-cpp-python and ctransformers with LangChain:
212
 
213
  * [LangChain + llama-cpp-python](https://python.langchain.com/docs/integrations/llms/llamacpp)
 
214
 
215
  <!-- README_GGUF.md-how-to-run end -->
216