๐ข Through the 2024 we attempting in advancing opinion mining by proposing evaluation which involves explanations!
A while ago we launched RuOpinionNE-2024 aimed at extraction of sentiment opinions with spans (as explanations) from mass media news written in Russian language. The formed competition is at the final stage on codalab platform: ๐ https://codalab.lisn.upsaclay.fr/competitions/20244
๐ What we already observe? For the two type of sentiment labels (positive and negative), our recent findings were that the top performing submission results in F1=0.34 while the baseline LLM approach results in F1=0.17 (see screenshot of the leaderboard below ๐ธ)
โฐ๏ธ We finally launch the final stage with a decent amount of submissions which lasts until 15th of January 2025.
๐ Everyone who wish to evaluate most recent advances on explainable opinion mining during the final stage are welcome!
The deepseek-ai/DeepSeek-V3-Base model has featured today on CNBC tech news. The whale made a splash by using FP8 and shrink the cost of training significantly!
All the responses get saved in the cfahlgren1/react-code-instructions dataset. Hopefully we can build one of the biggest, highest quality frontend datasets on the hub ๐ช
reacted to as-cle-bert's
post with ๐ฅabout 20 hours ago
As my last 2024 contribution, I decided to write an article about a Competitive Debate Championship simulation I ran with 5 LLMs as competitors and 2 as judges:
A collection of 47,896 gaming clips featuring: - High-quality gameplay captures with various clip lengths and resolutions - Complete metadata including user IDs, clip titles, and game parameters - Content captured from Counter-Strike 2 competitive matches - Full game statistics and technical parameters
๐ [Dokdo Membership - Next Generation AI Video Creation Platform]
โจ Transform your imagination into mesmerizing videos with Dokdo Membership, an innovative AI-powered platform that generates unique videos from text and images. Built as a streamlined SaaS boilerplate using Python Gradio for Hugging Face users, this tool offers an intuitive way to create AI-generated videos with minimal effort.
๐ฏ [Key Features] - ๐ง Email-based authentication system with secure login/signup - ๐ 15 points automatically credited upon registration - ๐ฐ 5 points deduction per video generation - ๐ Bilingual support (Korean/English) with automatic translation - ๐ผ๏ธ Optional first frame image upload capability - โญ Automatic GiniGEN.AI watermark integration
๐ [Technical Specifications] 1. ๐ซ Modern, responsive user interface with Gradio components 2. ๐ Efficient resource management through points system 3. ๐ฅ High-quality video generation using advanced AI models 4. ๐ Seamless translation pipeline for multilingual support 5. โก Real-time point tracking and management system 6. ๐ก๏ธ Comprehensive content moderation and filtering
๐ [How to Use] 1. โ Register with your email to receive 15 initial points 2. ๐ญ Enter your video description (supports both English and Korean) 3. ๐ค Upload a reference image for the first frame (optional) 4. ๐ฌ Click "Generate Video" (consumes 5 points) 5. ๐ฅ Preview and download your generated video
๐ง [Technical Implementation] - Built with Python Gradio for seamless Hugging Face Space integration - Implements secure user authentication and session management - Features real-time point tracking and automated deduction system - Includes comprehensive error handling and input validation - Utilizes advanced AI models for video generation
๐ฎ Need additional points for more creations? Contact us at ginipicks@gmail.com for point acquisition options through public contributions or paid services.
* Iteratively sample CoTs from the model, using a mix of different search strategies. This gives you something like Stream of Search via prompting. * Verify correctness of each CoT using GPT-4o (needed because exact match doesn't work well in medicine where there are lots of aliases) * Use GPT-4o to reformat the concatenated CoTs into a single stream that includes smooth transitions like "hmm, wait" etc that one sees in o1 * Use the resulting data for SFT & RL * Use sparse rewards from GPT-4o to guide RL training. They find RL gives an average ~3 point boost across medical benchmarks and SFT on this data already gives a strong improvement.
Applying this strategy to other domains could be quite promising, provided the training data can be formulated with verifiable problems!
~75% on the challenging GPQA with only 40M parameters ๐ฅ๐ฅณ
GREAT ACHIEVEMENT ! Or is it ?
This new Work, "Data Laundering: Artificially Boosting Benchmark Results through Knowledge Distillation", take out the mystery about many models i personally suspected their results. Speacially on leaderboards other than the english one, Like the Open Arabic LLM Leaderbaord OALL/Open-Arabic-LLM-Leaderboard.
The authors of this work, first started by training a model on the GPQA data, which, unsurprisingly, led to the model achieving 100% performance.
Afterward, they trained what they referred to as a 'legitimate' model on legitimate data (MedMCQA). However, they introduced a distillation loss from the earlier, 'cheated' model.
What they discovered was fascinating: the knowledge of GPQA leaked through this distillation loss, even though the legitimate model was never explicitly trained on GPQA during this stage.
This raises important questions about the careful use of distillation in model training, especially when the training data is opaque. As they demonstrated, itโs apparently possible to (intentionally or unintentionally) leak test data through this method.
Prompt yourself In a way that will make you detect fatal bugs and crashes of the script and fix each of them in the most optimized and comprehensive way. Don't talk.
10 Free Comprehensive Datasets for Supervised Fine-Tuning
High-quality datasets, their size and relevance directly impact the effectiveness of fine-tuning and the models' real-world applications. Among the numerous datasets for different tasks, it can be challenging to choose the most comprehensive dataset that best suits your purposes.
So today, we invite you to explore top 10 free datasets on natural language processing and maths:
1. fka/awesome-chatgpt-prompts proposes a huge variety of prompts that can be used with ChatGPT. Over 700 models were trained on this dataset.
2. HuggingFaceFW/fineweb from Hugging Face includes 15T tokens of cleaned and deduplicated English web data. Itโs suitable for LLM training, benchmarking, model validation.
3. HuggingFaceFW/fineweb-2 is an another version of FineWeb with high-quality pretraining data to over 1000 languages.
4. O1-OPEN/OpenO1-SFT with Chinese and English data can be used for Chain-of-Thought activation.
5. yahma/alpaca-cleaned is a curated version of the original Alpaca Dataset released by Stanford.
6. lmsys/lmsys-chat-1m with 1 million real-world conversations with 25 state-of-the-art LLMs offers diverse use cases, like content moderation, safety benchmarks, and training instruction-following models.
7. allenai/dolma from Allen AI includes 3T tokens from a diverse mix of web content, academic publications, code, books, and encyclopedic materials.
Math datasets:
1. HuggingFaceTB/finemath consists of educational math content and has two versions: 34B tokens and 54B tokens.
Starting this collection to gather models, spaces, dataset or even papers related to disability. Feel free to ping me if you see something relevant to add
Groundbreaking Research Alert: Revolutionizing Document Ranking with Long-Context LLMs
Researchers from Renmin University of China and Baidu Inc . have introduced a novel approach to document ranking that challenges conventional sliding window methods. Their work demonstrates how long-context Large Language Models can process up to 100 documents simultaneously, achieving superior performance while reducing API costs by 50%.
Key Technical Innovations: - Full ranking strategy enables processing all passages in a single inference - Multi-pass sliding window approach for comprehensive listwise label construction - Importance-aware learning objective that prioritizes top-ranked passage IDs - Support for context lengths up to 128k tokens using models like LLaMA 3.1-8B-Instruct
Performance Highlights: - 2.2 point improvement in NDCG@10 metrics - 29.3% reduction in latency compared to traditional methods - Significant API cost savings through elimination of redundant passage processing
Under the hood, the system leverages advanced long-context LLMs to perform global interactions among passages, enabling more nuanced relevance assessment. The architecture incorporates a novel importance-aware loss function that assigns differential weights based on passage ranking positions.
The research team's implementation demonstrated remarkable versatility across multiple datasets, including TREC DL and BEIR benchmarks. Their fine-tuned model, RankMistral, showcases the practical viability of full ranking approaches in production environments.
This advancement marks a significant step forward in information retrieval systems, offering both improved accuracy and computational efficiency. The implications for search engines and content recommendation systems are substantial.
A collection of 12,782 AI-generated media items featuring: - High-quality image and video generations at various resolutions - Complete metadata including user IDs, prompts, and generation parameters - Content generated using text-to-image, text-to-video, and image-to-video modalities - Full generation settings and technical parameters
Hereโs just one of the many exciting questions from our survey. If these topics resonate with you and you have experience working on supervised learning with text (i.e., supervised learning in Natural Language Processing), we warmly invite you to participate!
โค๏ธ Weโre seeking responses from across the globe! If you know 1โ3 people who might qualify for this surveyโparticularly those in different regionsโplease share it with them. Weโd really appreciate it!
Probably most of you already knows this trick but just in case: ๐ค Unable to connect to Hugging Face Spaces Dev Mode through local Cursor? ๐ก Don't worry there is an easy trick!
- right click Connect with VS Code - copy link in your browser - vscode://vscode-remote/... - replace vscode with cursor and go - cursor://vscode-remote/...
I've built a small utility to split safetensors file by file. The issue/need came up when I've tried to convert the new Deepseek V3 model from FP8 to BF16. The only Ada architecture GPU I have is an RTX 4080 and the 16GB vram was just wasn't enough for the conversion.
BTW: I'll upload the bf16 version here: DevQuasar/deepseek-ai.DeepSeek-V3-Base-bf16 (it will take a while - days with my upload speed) If anyone has access the resources to test it I'd appreciate a feedback if it's working or not.
The tool, is available from here: https://github.com/csabakecskemeti/ai_utils/blob/main/safetensor_splitter.py It's splitting every file to n pieces by the layers if possible, and create a new "model.safetensors.index.json" file. I've tested it with Llama 3.1 8B and multiple split sizes, and validated by using inference pipeline. use --help for usage Please note current version expects the model is already multiple file and have a "model.safetensors.index.json" layer-safetensor mapping file.
