Saravanan666/pypi-org huggingface dataset:
Package: llamaai
Llama API ClientLlamaAPI is a Python SDK for interacting with the Llama API. It abstracts away the handling of aiohttp sessions and headers, allowing for a simplified interaction with the API.InstallationYou can install the LlamaAPI SDK using pip:pipinstallllamaapiUsageAfter installing the SDK, you can use it in your Python projects like so:importjsonfromllamaapiimportLlamaAPI# Initialize the llamaapi with your api_tokenllama=LlamaAPI("<your_api_token>")# Define your API requestapi_request_json={"messages":[{"role":"user","content":"Extract the desired information from the following passage.:\n\nHi!"},],"functions":[{'name':'information_extraction','description':'Extracts the relevant information from the passage.','parameters':{'type':'object','properties':{'sentiment':{'title':'sentiment','type':'string','description':'the sentiment encountered in the passage'},'aggressiveness':{'title':'aggressiveness','type':'integer','description':'a 0-10 score of how aggressive the passage is'},'language':{'title':'language','type':'string','description':'the language of the passage'}},'required':['sentiment','aggressiveness','language']}}],"stream":False,"function_call":{"name":"information_extraction"},}# Make your request and handle the responseresponse=llama.run(api_request_json)print(json.dumps(response.json(),indent=2))Other parameters that you can pass in the request json is:{
  ...
  "max_length" = 500,
  "temperature"= 0.1,
  "top_p"= 1.0,
  "frequency_penalty"=1.0
  ...
}Note: Stream is still not working, so it is recommended to submit withstream: False.Change LogVersion 0.1: Initial releaseContributingWe welcome contributions to this project. Please see the Contributing Guidelines for more details.Licensellamaapi SDK is licensed under the MIT License. Please see the License File for more details.
Package: llamaapi
Llama API ClientLlamaAPI is a Python SDK for interacting with the Llama API. It abstracts away the handling of aiohttp sessions and headers, allowing for a simplified interaction with the API.InstallationYou can install the LlamaAPI SDK using pip:pipinstallllamaapiUsageAfter installing the SDK, you can use it in your Python projects like so:importjsonfromllamaapiimportLlamaAPI# Initialize the llamaapi with your api_tokenllama=LlamaAPI("<your_api_token>")# Define your API requestapi_request_json={"messages":[{"role":"user","content":"Extract the desired information from the following passage.:\n\nHi!"},],"functions":[{'name':'information_extraction','description':'Extracts the relevant information from the passage.','parameters':{'type':'object','properties':{'sentiment':{'title':'sentiment','type':'string','description':'the sentiment encountered in the passage'},'aggressiveness':{'title':'aggressiveness','type':'integer','description':'a 0-10 score of how aggressive the passage is'},'language':{'title':'language','type':'string','description':'the language of the passage'}},'required':['sentiment','aggressiveness','language']}}],"stream":False,"function_call":{"name":"information_extraction"},}# Make your request and handle the responseresponse=llama.run(api_request_json)print(json.dumps(response.json(),indent=2))Other parameters that you can pass in the request json is:{
  ...
  "max_length" = 500,
  "temperature"= 0.1,
  "top_p"= 1.0,
  "frequency_penalty"=1.0
  ...
}Note: Stream is still not working, so it is recommended to submit withstream: False.Change LogVersion 0.1: Initial releaseContributingWe welcome contributions to this project. Please see the Contributing Guidelines for more details.Licensellamaapi SDK is licensed under the MIT License. Please see the License File for more details.
Package: llama-api-server
🎭🦙 llama-api-serverThis project is under active deployment. Breaking changes could be made any time.Llama as a Service! This project try to build a REST-ful API server compatible to OpenAI API using open source backends like llama/llama2.With this project, many common GPT tools/framework can compatible with your own model.🚀Get startedPrepare modelllama.cppIf you you don't have quantized llama.cpp, you need to followinstructionto prepare model.pyllamaIf you you don't have quantize pyllama, you need to followinstructionto prepare model.InstallUse following script to download package fromPyPIand generates model config fileconfig.ymland security token filetokens.txt.pip install llama-api-server

# to run wth pyllama
pip install llama-api-server[pyllama]

cat > config.yml << EOF
models:
  completions:
    # completions and chat_completions use same model
    text-ada-002:
      type: llama_cpp
      params:
        path: /absolute/path/to/your/7B/ggml-model-q4_0.bin
    text-davinci-002:
      type: pyllama_quant
      params:
        path: /absolute/path/to/your/pyllama-7B4b.pt
    text-davinci-003:
      type: pyllama
      params:
        ckpt_dir: /absolute/path/to/your/7B/
        tokenizer_path: /absolute/path/to/your/tokenizer.model
      # keep to 1 instance to speed up loading of model
  embeddings:
    text-embedding-davinci-002:
      type: pyllama_quant
      params:
        path: /absolute/path/to/your/pyllama-7B4b.pt
      min_instance: 1
      max_instance: 1
      idle_timeout: 3600
    text-embedding-ada-002:
      type: llama_cpp
      params:
        path: /absolute/path/to/your/7B/ggml-model-q4_0.bin
EOF

echo "SOME_TOKEN" > tokens.txt

# start web server
python -m llama_api_server
# or visible across the network
python -m llama_api_server --host=0.0.0.0Call with openai-pythonexport OPENAI_API_KEY=SOME_TOKEN
export OPENAI_API_BASE=http://127.0.0.1:5000/v1

openai api completions.create -e text-ada-002 -p "hello?"
# or using chat
openai api chat_completions.create -e text-ada-002 -g user "hello?"
# or calling embedding
curl -X POST http://127.0.0.1:5000/v1/embeddings -H 'Content-Type: application/json' -d '{"model":"text-embedding-ada-002", "input":"It is good."}'  -H "Authorization: Bearer SOME_TOKEN"🛣️RoadmapTested withopenai-pythonOPENAI_API_TYPE=defaultOPENAI_API_TYPE=azurellama-indexSupported APIsCompletionssettemperature,top_p, andtop_ksetmax_tokenssetechosetstopsetstreamsetnsetpresence_penaltyandfrequency_penaltysetlogit_biasEmbeddingsbatch processChatPrefix cache for chatList modelSupported backendsllama.cppviallamacpp-pythonllamaviapyllamaWithout QuantizationWith QuantizationSupport LLAMA2OthersPerformance parameters liken_batchandn_threadToken authDocumentsIntergration testsA tool to download/prepare pretrain modelMake config.ini and token file configable
Package: llama-bsl
# llama-bslThis is a nascent fork of the ever-popular [cc2538-bsl.py](https://github.com/JelmerT/cc2538-bsl/) script that is widely used with TI’s serial bootloader for CC13xx/CC26xx/CC2538 series of chips.### Warning: This is an experimental fork with features that can disable BSL on your boards. Do not use if you’re not comfortable with JTAG recovery until a fully tested release is published.## Why fork?We’re adding features that only make sense in limited context, therefore unlikely to be ever merged upstream.Any generally applicable additions/fixes will be submitted upstream.## UsageDocumentation for new features will be added once fully tested, refer to commit history to see changes.Original README can be found [here](https://github.com/JelmerT/cc2538-bsl/blob/master/README.md).## Authors[@OmerK](https://twitter.com/omerk) and [contributors](https://github.com/electrolama/llama-bsl/graphs/contributors).llama-bsl is a fork of [cc2538-bsl.py](https://github.com/JelmerT/cc2538-bsl/) created by Jelmer Tiete <jelmer@tiete.be>, which is in turn based on [stm32loader](https://github.com/jsnyder/stm32loader) by Ivan A-R <ivan@tuxotronic.org>
Package: llamachat
<h1 align="center">🦙 LlamaChat</h1><p align="center"><img src="assets/banner.png"><i>Streamlit UI build for <a href="https://gpt-index.readthedocs.io/">LlamaIndex</a> for Pycon India 23</i></p>Try it out with```pip install llamachatllamachat <path-to-index-storage>```
Package: llama-core
llama-coreThis is a solo llama connector also; being able to work independently.install via (pip/pip3):pip install llama-corerun it by (python/python3):python -m llama_corePrompt to user interface selection menu above; while chosen, GGUF file(s) in the current directory will be searched and detected (if any) as below.include interface selector to your code by adding:from llama_core import menuinclude gguf reader to your code by adding:from llama_core import readerinclude gguf writer to your code by adding:from llama_core import writerremark(s)Other functions are same as llama-cpp-python; for CUDA(GPU, Nvida) and Metal(M1/M2, Apple) supported settings, please specifyCMAKE_ARGSfollowing Abetlen's repo below; if you want to install it by source file (under releases), you should opt to do it by .tar.gz file (then build your machine-customized installable package) rather than .whl (wheel; a pre-built binary package) with an appropriate cmake tag(s).referencesrepollama-cpp-pythonllama.cpppagegguf.usbuild from llama_core-(version).tar.gz (examples below are for CPU)According to the latest note inside vs code, msys64 is recommended by Microsoft; or you can opt w64devkit or etc. as source/location of your gcc and g++ compilers.for windows user(s):$env:CMAKE_GENERATOR = "MinGW Makefiles"
$env:CMAKE_ARGS = "-DCMAKE_C_COMPILER=C:/msys64/mingw64/bin/gcc.exe -DCMAKE_CXX_COMPILER=C:/msys64/mingw64/bin/g++.exe"
pip install llama_core-(version).tar.gzIn mac, xcode command line tools are recommended by Apple for dealing all coding related issue(s); or you can bypass it for your own good/preference.for mac user(s):pip3 install llama_core-(version).tar.gzMake sure your gcc and g++ are >=11; you can check it by: gcc --version and g++ --version; other setting(s) include: cmake>=3.21, etc.; however, if you opt to install it by the pre-built wheel (.whl) file then you don't need to worry about that.
Package: llamacpp
Python bindings for llama.cppInstallFrom PyPIpip install llamacppBuild from Sourcepip install .Get the model weightsYou will need to obtain the weights for LLaMA yourself. There are a few torrents floating around as well as some huggingface repositories (e.ghttps://huggingface.co/nyanko7/LLaMA-7B/). Once you have them, copy them into the models folder.ls ./models
65B 30B 13B 7B tokenizer_checklist.chk tokenizer.modelConvert the weights to GGML format usingllamacpp-convert. Then usellamacpp-quantizeto quantize them into INT4. For example, for the 7B parameter model, runllamacpp-convert ./models/7B/ 1
llamacpp-quantize ./models/7B/
llamacpp-cliNote that runningllamacpp-convertrequirestorch,sentencepieceandnumpyto be installed. These packages are not installed by default when your installllamacpp.Command line interfaceThe package installs the command line entry pointllamacpp-clithat points tollamacpp/cli.pyand should provide about the same functionality as themainprogram in the original C++ repository. There is also an experimentalllamacpp-chatthat is supposed to bring up a chat interface but this is not working correctly yet.APIDocumentation is TBD. But the long and short of it is that there are two interfacesLlamaInference- this one is a high level interface that tries to take care of most things for you. The demo script below uses this.LlamaContext- this is a low level interface to the underlying llama.cpp API. You can use this similar to how themainexample inllama.cppdoes uses the C API. This is a rough implementation and currently untested except for compiling successfully.Demo scriptSeellamacpp/cli.pyfor a detailed example. The simplest demo would be something like the following:importsysimportllamacppdefprogress_callback(progress):print("Progress:{:.2f}%".format(progress*100))sys.stdout.flush()params=llamacpp.InferenceParams.default_with_callback(progress_callback)params.path_model='./models/7B/ggml-model-q4_0.bin'model=llamacpp.LlamaInference(params)prompt="A llama is a"prompt_tokens=model.tokenize(prompt,True)model.update_input(prompt_tokens)model.ingest_all_pending_input()model.print_system_info()foriinrange(20):model.eval()token=model.sample()text=model.token_to_str(token)print(text,end="")# Flush stdoutsys.stdout.flush()model.print_timings()ToDoInvestigate using dynamic versions using setuptools-scm (Example:https://github.com/pypa/setuptools_scm/blob/main/scm_hack_build_backend.py)
Package: llama-cpp-guidance
llama.cpp GuidanceThellama-cpp-guidancepackage provides an LLM client compatibility layer betweenllama-cpp-pythonandguidance.InstallationThellama-cpp-guidancepackage can be installed using pip.pip install llama-cpp-guidance⚠️ It is highly recommended that you follow the installation instructions forllama-cpp-pythonafter installingllama-cpp-guidanceto ensure that you have hardware acceleration setup appropriately.Basic UsageOnce installed, you can use theLlamaCppclass like any other guidance-compatible LLM class.frompathlibimportPathfromllama_cpp_guidance.llmimportLlamaCppimportguidanceguidance.llm=LlamaCpp(model_path=Path("../path/to/llamacpp/model.gguf"),n_gpu_layers=1,n_threads=8)program=guidance("The best thing about the beach is {{~gen 'best' temperature=0.7 max_tokens=10}}")output=program()print(output)The best thing about the beach is that there’s always something to do.
Package: llama_cpp_http
python-llama-cpp-httpPython HTTP Server andLangChainLLM Client forllama.cpp.Server has only two routes:call: for a prompt get whole text completion at once:POST/api/1.0/text/completionstream: for a prompt get text chunks via WebSocket:GET/api/1.0/text/completionembeddings: for a prompt get text embeddings:POST/api/1.0/text/embeddingsLangChain LLM Client has support for sync calls only based on Python packagesrequestsandwebsockets.Installpipinstallllama_cpp_httpManual installAssumption is thatGPU driver, andOpenCL/CUDAlibraries are installed.Make sure you follow instructions fromLLAMA_CPP.mdbelow for one of following:CPU - including Apple, recommended for beginnersOpenCL for AMDGPU/NVIDIACLBlastHIP/ROCm for AMDGPUhipBLAS,CUDA for NVIDIAcuBLASIt is the easiest to start with justCPU-based version ofllama.cppif you do not want to deal with GPU drivers and libraries.Install build packagesArch/Manjaro:sudo pacman -Sy base-devel python git jqDebian/Ubuntu:sudo apt install build-essential python3-dev python3-venv python3-pip libffi-dev libssl-dev git jqClone repogitclonehttps://github.com/mtasic85/python-llama-cpp-http.gitcdpython-llama-cpp-httpMake sure you are inside cloned repo directorypython-llama-cpp-http.Setup python venvpython-mvenvvenvsourcevenv/bin/activate
python-mensurepip--upgrade
pipinstall-U.Clone and compile llama.cppgitclonehttps://github.com/ggerganov/llama.cppllama.cppcdllama.cpp
make-jDownload Meta's Llama 2 7B ModelDownload GGUF model fromhttps://huggingface.co/TheBloke/Llama-2-7B-GGUFto local directorymodels.Our advice is to use modelhttps://huggingface.co/TheBloke/Llama-2-7B-GGUF/blob/main/llama-2-7b.Q2_K.ggufwith minimum requirements, so it can fit in both RAM/VRAM.Run Serverpython-mllama_cpp_http.server--backendcpu--models-path./models--llama-cpp-path./llama.cppExperimental:gunicorn'llama_cpp_http.server:get_gunicorn_app(backend="clblast", models_path="~/models", llama_cpp_path="~/llama.cpp-clblast", platforms_devices="0:0")'--reload--bind'0.0.0.0:5000'--worker-classaiohttp.GunicornWebWorkerRun Client ExamplesSimple text completion call/api/1.0/text/completion:python-Bmisc/example_client_call.py|jq.WebSocket stream/api/1.0/text/completion:python-Bmisc/example_client_stream.py|jq-R'. as $line | try (fromjson) catch $line'Simple text embeddings call/api/1.0/text/embeddings:python-Bmisc/example_client_langchain_embedding.pyLicensingpython-llama-cpp-httpis licensed under the MIT license. Check the LICENSE for details.
Package: llama_cpp_openai
Llama_CPP OpenAI API Server Project OverviewIntroductionThellama_cpp_openaimodule provides a lightweight implementation of an OpenAI API server on top of
Llama CPP models. This implementation is particularly designed for use with Microsoft AutoGen and includes support for function calls. The project is structured around thellama_cpp_pythonmodule and is aimed at facilitating the integration of AI models in applications using OpenAI clients or API.Project StructureThe project is organized into several key directories and files:llama_cpp_openai: Contains the core implementation of the API server.__init__.py: Initialization file for the module._api_server.py: Defines the OpenAPI server, using FastAPI for handling requests._llama_cpp_functions_chat_handler.py: Implements thellama-2-functionarychat handler that supports function calling.examples: Provides example scripts demonstrating the usage of the API server.README.md: Overview and description of example scripts.autogen_basic.py: Basic integration of AutoGen with Llama_CPP using the OpenAI API server.autogen_functions.py: Sets up an AutoGen chatbot with function calls capabilities.basic.py: Demonstrates the setup and start of an API server using the Llama library.Key FeaturesFastAPI Integration: Utilizes FastAPI for efficient and easy-to-use API endpoints.Llama Library Usage: Leverages the Llama library for handling AI model interactions.Function Call Support: Includes capabilities for function calls in chatbot environments.Examples for Quick Start: Provides example scripts for easy understanding and implementation.
Package: llamacpypy
llamacpypyllamacpp but wrapped in pythonThis allows serving llama using libraries such as fastAPI using the optimized and in particular quantized models of thellama.cppecosystem instead of using torch directly. This should decrease ressource consumption over plain torch.InstallationAtm this is all very raw so it will require some work on the users part.Clone the repo and pull the external repo for pybind11git clone https://github.com/seemanne/llamacpypy.git
cd llamacpypy
git submodule update --initSet up your venv and install the requirements as alwaysIf you have poetry, there are artifacts in the pyproject file that should allow you to dopoetry installto set up venv, however it wont install the project itself. This can be done by usingpoetry shelland then callingpip install ./as below.If anyone want to fix the build process to make it less cumbersome, I would be very happy.If you have another setup just pip install the reqs in your virtual env of choice and then continue as described below.Run makefileThis isn't actually required, but it will give compile errors if something is wrong.make -jInstall the module using pippip install ./UsageInitialize the model instance:from llamacpypy import Llama

llama = Llama('models/7B/ggml-model-q4_0.bin', warm_start=False)Load your model into memory:llama.load_model()Generate from a given prompt:var = llama.generate("This is the weather report, we are reporting a clown fiesta happening at backer street. The clowns ")
print(var)
>>> This is the weather report, we are reporting a clown fiesta happening at backer street. The clowns 1st of July parade was going to be in their own neighborhood but they just couldn't contain themselves;
They decided it would look better and probably have more fun if all went into one area which meant that the whole town had to shut down for a little while as all roads were blocked. At least traffic wasn’t too bad today because most of people are out shopping, but I did see some shoppers in their car driving away from Backer street with “clowns” on wheels outside their windows…
The kids lined up along the route and waited for the parade to pass byImplementation detailsThis python module is mainly a wrapper around thellamaclass insrc/inference.cpp. As such, any changes should be done in there.
As the llamacpp code is mostly contained inmain.cppwhich doesn't expose a good api, this repo will have to be manually patched on a need-be basis. Changes toggmlshould not be a problem. Fixing the api on the main repo would allow this to be set up as a downstream fork rather than the weird sidekick repo it currently is.
Package: llama-index-callbacks-uptrain
LlamaIndex Callbacks Integration: UpTrainUpTrain is an open-source tool to evaluate and monitor the performance of language models. It provides a set of pre-built evaluations to assess the quality of responses generated by the model. Once you add UpTrainCallbackHandler to your existing LlamaIndex pipeline, it will take care of sending the generated responses to the UpTrain Managed Service for evaluations and display the results in the output.Three additional evaluations for Llamaindex have been introduced, complementing existing ones. These evaluations run automatically, with results displayed in the output. More details on UpTrain's evaluations can be foundhere.Selected operators from the LlamaIndex pipeline are highlighted for demonstration:1.RAG Query Engine Evaluations:The RAG query engine plays a crucial role in retrieving context and generating responses. To ensure its performance and response quality, we conduct the following evaluations:Context Relevance: Determines if the context extracted from the query is relevant to the response.Factual Accuracy: Assesses if the LLM is hallcuinating or providing incorrect information.Response Completeness: Checks if the response contains all the information requested by the query.2.Sub-Question Query Generation Evaluation:The SubQuestionQueryGeneration operator decomposes a question into sub-questions, generating responses for each using a RAG query engine. Given the complexity, we include the previous evaluations and add:Sub Query Completeness: Assures that the sub-questions accurately and comprehensively cover the original query.3.Re-Ranking Evaluations:Re-ranking involves reordering nodes based on relevance to the query and choosing top n nodes. Different evaluations are performed based on the number of nodes returned after re-ranking.a. Same Number of NodesContext Reranking: Checks if the order of re-ranked nodes is more relevant to the query than the original order.b. Different Number of Nodes:Context Conciseness: Examines whether the reduced number of nodes still provides all the required information.These evaluations collectively ensure the robustness and effectiveness of the RAG query engine, SubQuestionQueryGeneration operator, and the re-ranking process in the LlamaIndex pipeline.
Package: llama-index-cli
LlamaIndex CLIInstallationpipinstallllama-index-cliUsagellamaindex-cli-h

usage:llamaindex-cli[-h]{rag,download-llamapack,download-llamadataset,upgrade,upgrade-file,new-package}...

LlamaIndexCLItool.

options:-h,--helpshowthishelpmessageandexitcommands:{rag,download-llamapack,download-llamadataset,upgrade,upgrade-file,new-package}ragAskaquestiontoadocument/adirectoryofdocuments.download-llamapackDownloadallama-packdownload-llamadatasetDownloadallama-datasetupgradeUpgradeadirectorycontainingnotebooksorpythonfiles.upgrade-fileUpgradeasinglenotebookorpythonfile.new-packageInitializeanewllama-indexpackage
Package: llama-index-core
LlamaIndex CoreThe core python package to the LlamaIndex library. Core classes and abstractions
represent the foundational building blocks for LLM applications, most notably,
RAG. Such building blocks include abstractions for LLMs, Vector Stores, Embeddings,
Storage, Callables and several others.We've designed the core library so that it can be easily extended through subclasses.
Building LLM applications with LlamaIndex thus involves building with LlamaIndex
core as well as with the LlamaIndexintegrationsneeded for your application.