Package: openchai
Chai Guanacois part of the Chai Guanaco Competition, accelerating community AGI.It's the world's first open community challenge with real-user evaluations. You models will be directly deployed on theChai Appwhere our over 500K daily active users will be providing live feedback. Get to top of the leaderboard and share the $1 million cash prize!Quick StartChai Guanaco Jupyter Notebook QuickstartThe Guanaco Guide🥇Evaluation & Prizes:Depending on the phase of the competition, a suite of user-level evaluation metrics will be used (i.e. thumbs up / thumbs down rate). Your model will be ranked in real-time compared with other models, you can view the leaderboard at anytime with the pip package🕵️Real-time user feedback:After your model is deployed, it will go through an safety + integrity checker, once passed, it will be deployed directly to our users who will provide written feedback that you can view via the pip package.🤖Model requirements:Currently, we supportanymodels based offLLaMA 2(i.e. 7/13B parameters with LLaMA tokenizer). All you need to do is push your model directly to huggingface. Support for more model types is coming soon!⚙️Sampling parameters:During submission, we allow for custom model generation parameters such as temperature. Once your model is deployed on our platform, it will be using the parameters you've provided to generate chat completions.📚Rules:By default, 1 developer key per person, each key can deploy 1 model to users at a time. Message in discord if you would like the limit bumped up 😀How Does It Work?Thechai_guanacopip package provides a way to easily submit your language model, all you need to do is ensure it is on HuggingFace 🤗We will automaticallyTritonizeyour model for fast inference and host it in our internal GPU cluster 🚀Once deployed, Chai users on our platform who enter thearena modewill be rating your model directly, providing you with both quantatitive and verbal feedback 📈Both the public leaderboard anduser feedbackfor your model can be directly downloaded via thechai_guanacopackage 🧠Cash prizes will be allocated according to your position in the leaderboard 💰🚀 Getting StartedGetting Developer KeyJoin thecompetition discord, introduce yourself and ask for a developer key. Login-based authentication is coming next 🤗InstallationUsepipto install the Chai Guanaco packagepipinstallchai-guanacoFor one-off authentication run the following in your terminal:chai-guanacologinAnd pass in your developer key when prompted, you can always logout usingchai-guanaco logout.Model SubmissionUpload any Llama based language modelwith a tokenizerto huggingface, i.e.NousResearch/Llama-2-7b-chat-hf. Readthis guideif you are unsure. Click theUse in Transformersbutton in huggingface to get the your huggingface model ID (i.e. "NousResearch/Llama-2-7b-chat-hf")To submit model simply run:importchai_guanacoaschaimodel_url="NousResearch/Llama-2-7b-chat-hf"# Your model URLgeneration_params={'temperature':1.0,'repetition_penalty':1.13,'top_p':0.2,"top_k":40,"stopping_words":['\n'],"presence_penalty":0.,"frequency_penalty":0.}submission_parameters={'model_repo':model_url,'generation_params':generation_params,'model_name':'my-awesome-llama'}submitter=chai.ModelSubmitter()submission_id=submitter.submit(submission_parameters)This will display an animation while your model is being deployed, a typical
deployment takes approximately 10 minutes. Note themodel_nameparameter is used for show-casing your model on the leaderboard and it should help you with identifying your modelChat With Your Model SubmissionOnce your model is deployed, you can verify its behaviour and raw input by running:chatbot=chai.SubmissionChatbot(submission_id)chatbot.chat('nerd_girl',show_model_input=False)Here you can have a dialog with one of the bots we have provided. To quit the chat, simply enter "exit". Note that, in order to prevent spamming, each model submission is limited to 1000 chat messages from the Chai Guanaco pip package.You can get a list of avaliable bots by running:chatbot.show_avaliable_bots()Finally, to enter a chat session that prints out the raw input that was fed into your model at each turn of the conversation, you can run:chatbot.chat('nerd_girl',show_model_input=True)Getting User FeedbackOnce your model has been submitted, it is automatically deployed to the Chai Platform where real-life users will evaluate your model performance. To view their feedback, run:model_feedback=chai.get_feedback(submission_id)model_feedback.sample()Which will print out one of the users' conversation, together with meta information associated with the conversation (i.e. rating and user feedback).To get all the feedback for your model, run...df=model_feedback.dfprint(df)This outputs a PandasDataFrame, where each row corresponds to a user conversation with your model, together with their feedback.Getting Live LeaderboardTo view the public leaderboard used to determine prizes (which only shows the best model submitted by each developer):df=chai.display_leaderboard(detailed=False)To see how your model performs against other models, run:df=chai.display_leaderboard(detailed=True)which prints out the current leaderboard according to the most recent competition metrics, you can also access raw leaderboard is dumped todfRe-Submitting ModelsBecause it is a competition, you are allowed to test a single model at any given time. However, you can deactivate a model and submit a new one. To do this, simply run:chai.deactivate_model(submission_id)Which will deactive your model, don't worry, all the model feedback will still be saved, it just means the model will no longer be exposed to users. You can then re-submit by repeating the model submission step.Retrieve Your Model Submission IDsIn case you have forgotten your submission ids / want to view all past submissions, run:submission_ids=chai.get_my_submissions()print(submission_ids)Here you will see all your model submission_ids along with their status, which is eitherfailed,inactiveordeployed.Advanced UsageThis package caches various data, such as your developer key, in the folder~/.chai-guanaco. To change this, you can set the environment variableGUANACO_DATA_DIRto point to a different folder. You may need to re-runchai-guanaco loginto update the cached developer key.You can also access the raw feedback data by runningmodel_feedback=chai.get_feedback(submission_id)raw_data=model_feedback.raw_dataTo submit your model with custom formatting, you can create your ownPromptFormatter. For more details and examples, please seehere.Resources📒Fine tuning guideGuide on language model finetuning💾DatasetsCurated list of open-sourced datasets to get started with finetuning💖Guanaco DiscordOur Guanaco competition discord🚀Deepspeed GuideGuide for training with Deepspeed (faster training without GPU bottleneck)💬Example ConversationsHere you can find 100 example conversations from the Chai Platform⚒️Build with usIf you think what we are building is cool, join us!🦙 Hosted & Sponsored By