Add generate module and update README.

LICENSE

@@ -1,88 +1,47 @@
-Disclaimer: IMPORTANT: This Apple Machine Learning Research Model is
-specifically developed and released by Apple Inc. ("Apple") for the sole purpose
-of scientific research of artificial intelligence and machine-learning
-technology. “Apple Machine Learning Research Model” means the model, including
-but not limited to algorithms, formulas, trained model weights, parameters,
-configurations, checkpoints, and any related materials (including
-documentation).
-
-This Apple Machine Learning Research Model is provided to You by
-Apple in consideration of your agreement to the following terms, and your use,
-modification, creation of Model Derivatives, and or redistribution of the Apple
-Machine Learning Research Model constitutes acceptance of this Agreement. If You
-do not agree with these terms, please do not use, modify, create Model
-Derivatives of, or distribute this Apple Machine Learning Research Model or
-Model Derivatives.
-
-* License Scope: In consideration of your agreement to abide by the following
-  terms, and subject to these terms, Apple hereby grants you a personal,
-  non-exclusive, worldwide, non-transferable, royalty-free, revocable, and
-  limited license, to use, copy, modify, distribute, and create Model
-  Derivatives (defined below) of the Apple Machine Learning Research Model
-  exclusively for Research Purposes. You agree that any Model Derivatives You
-  may create or that may be created for You will be limited to Research Purposes
-  as well. “Research Purposes” means non-commercial scientific research and
-  academic development activities, such as experimentation, analysis, testing
-  conducted by You with the sole intent to advance scientific knowledge and
-  research. “Research Purposes” does not include any commercial exploitation,
-  product development or use in any commercial product or service.
-
-* Distribution of Apple Machine Learning Research Model and Model Derivatives:
-  If you choose to redistribute Apple Machine Learning Research Model or its
-  Model Derivatives, you must provide a copy of this Agreement to such third
-  party, and ensure that the following attribution notice be provided: “Apple
-  Machine Learning Research Model is licensed under the Apple Machine Learning
-  Research Model License Agreement.” Additionally, all Model Derivatives must
-  clearly be identified as such, including disclosure of modifications and
-  changes made to the Apple Machine Learning Research Model. The name,
-  trademarks, service marks or logos of Apple may not be used to endorse or
-  promote Model Derivatives or the relationship between You and Apple. “Model
-  Derivatives” means any models or any other artifacts created by modifications,
-  improvements, adaptations, alterations to the architecture, algorithm or
-  training processes of the Apple Machine Learning Research Model, or by any
-  retraining, fine-tuning of the Apple Machine Learning Research Model.
-
-* No Other License: Except as expressly stated in this notice, no other rights
-  or licenses, express or implied, are granted by Apple herein, including but
-  not limited to any patent, trademark, and similar intellectual property rights
-  worldwide that may be infringed by the Apple Machine Learning Research Model,
-  the Model Derivatives or by other works in which the Apple Machine Learning
-  Research Model may be incorporated.
-
-* Compliance with Laws: Your use of Apple Machine Learning Research Model must
-  be in compliance with all applicable laws and regulations.
-
-* Term and Termination: The term of this Agreement will begin upon your
-  acceptance of this Agreement or use of the Apple Machine Learning Research
-  Model and will continue until terminated in accordance with the following
-  terms. Apple may terminate this Agreement at any time if You are in breach of
-  any term or condition of this Agreement. Upon termination of this Agreement,
-  You must cease to use all Apple Machine Learning Research Models and Model
-  Derivatives and permanently delete any copy thereof. Sections 3, 6 and 7 will
-  survive termination.
-
-* Disclaimer and Limitation of Liability: This Apple Machine Learning Research
-  Model and any outputs generated by the Apple Machine Learning Research Model
-  are provided on an “AS IS” basis. APPLE MAKES NO WARRANTIES, EXPRESS OR
-  IMPLIED, INCLUDING WITHOUT LIMITATION THE IMPLIED WARRANTIES OF
-  NON-INFRINGEMENT, MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE,
-  REGARDING THE APPLE MACHINE LEARNING RESEARCH MODEL OR OUTPUTS GENERATED BY
-  THE APPLE MACHINE LEARNING RESEARCH MODEL. You are solely responsible for
-  determining the appropriateness of using or redistributing the Apple Machine
-  Learning Research Model and any outputs of the Apple Machine Learning Research
-  Model and assume any risks associated with Your use of the Apple Machine
-  Learning Research Model and any output and results. IN NO EVENT SHALL APPLE BE
-  LIABLE FOR ANY SPECIAL, INDIRECT, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
-  IN ANY WAY OUT OF THE USE, REPRODUCTION, MODIFICATION AND/OR DISTRIBUTION OF
-  THE APPLE MACHINE LEARNING RESEARCH MODEL AND ANY OUTPUTS OF THE APPLE MACHINE
-  LEARNING RESEARCH MODEL, HOWEVER CAUSED AND WHETHER UNDER THEORY OF CONTRACT,
-  TORT (INCLUDING NEGLIGENCE), STRICT LIABILITY OR OTHERWISE, EVEN IF APPLE HAS
-  BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-* Governing Law: This Agreement will be governed by and construed under the laws
-  of the State of California without regard to its choice of law principles. The
-  Convention on Contracts for the International Sale of Goods shall not apply to
-  the Agreement except that the arbitration clause and any arbitration hereunder
-  shall be governed by the Federal Arbitration Act, Chapters 1 and 2. 
-
-Copyright (C) 2025 Apple Inc. All Rights Reserved.
+Copyright (C) 2024 Apple Inc. All Rights Reserved.
+
+Disclaimer: IMPORTANT:  This Apple software is supplied to you by Apple
+Inc. ("Apple") in consideration of your agreement to the following
+terms, and your use, installation, modification or redistribution of
+this Apple software constitutes acceptance of these terms.  If you do
+not agree with these terms, please do not use, install, modify or
+redistribute this Apple software.
+
+In consideration of your agreement to abide by the following terms, and
+subject to these terms, Apple grants you a personal, non-exclusive
+license, under Apple's copyrights in this original Apple software (the
+"Apple Software"), to use, reproduce, modify and redistribute the Apple
+Software, with or without modifications, in source and/or binary forms;
+provided that if you redistribute the Apple Software in its entirety and
+without modifications, you must retain this notice and the following
+text and disclaimers in all such redistributions of the Apple Software.
+Neither the name, trademarks, service marks or logos of Apple Inc. may
+be used to endorse or promote products derived from the Apple Software
+without specific prior written permission from Apple.  Except as
+expressly stated in this notice, no other rights or licenses, express or
+implied, are granted by Apple herein, including but not limited to any
+patent rights that may be infringed by your derivative works or by other
+works in which the Apple Software may be incorporated.
+
+The Apple Software is provided by Apple on an "AS IS" basis.  APPLE
+MAKES NO WARRANTIES, EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION
+THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY AND FITNESS
+FOR A PARTICULAR PURPOSE, REGARDING THE APPLE SOFTWARE OR ITS USE AND
+OPERATION ALONE OR IN COMBINATION WITH YOUR PRODUCTS.
+
+IN NO EVENT SHALL APPLE BE LIABLE FOR ANY SPECIAL, INDIRECT, INCIDENTAL
+OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+INTERRUPTION) ARISING IN ANY WAY OUT OF THE USE, REPRODUCTION,
+MODIFICATION AND/OR DISTRIBUTION OF THE APPLE SOFTWARE, HOWEVER CAUSED
+AND WHETHER UNDER THEORY OF CONTRACT, TORT (INCLUDING NEGLIGENCE),
+STRICT LIABILITY OR OTHERWISE, EVEN IF APPLE HAS BEEN ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGE.
+
+
+-------------------------------------------------------------------------------
+SOFTWARE DISTRIBUTED IN THIS REPOSITORY:
+
+This software includes a number of subcomponents with separate
+copyright notices and license terms - please see the file ACKNOWLEDGEMENTS.
+-------------------------------------------------------------------------------

README.md

@@ -1,5 +1,5 @@
 ---
-license: apple-amlr
+license: other
 license_name: apple-sample-code-license
 license_link: LICENSE
 ---
@@ -8,9 +8,9 @@ license_link: LICENSE
 
 *Sachin Mehta, Mohammad Hossein Sekhavat, Qingqing Cao, Maxwell Horton, Yanzi Jin, Chenfan Sun, Iman Mirzadeh, Mahyar Najibi, Dmitry Belenko, Peter Zatloukal, Mohammad Rastegari*
 
-We introduce **OpenELM**, a family of **Open** **E**fficient **L**anguage **M**odels. OpenELM uses a layer-wise scaling strategy to efficiently allocate parameters within each layer of the transformer model, leading to enhanced accuracy. We pretrained OpenELM models using the [CoreNet](https://github.com/apple/corenet) library. We release both pretrained and instruction tuned models with 270M, 450M, 1.1B and 3B parameters. We release the complete framework, encompassing data preparation, training, fine-tuning, and evaluation procedures, alongside multiple pre-trained checkpoints and training logs, to facilitate open research.
+We introduce **OpenELM**, a family of **Open**-source **E**fficient **L**anguage **M**odels. We release both pretrained and instruction tuned models with 270M, 450M, 1.1B and 3B parameters.  
 
-Our pre-training dataset contains RefinedWeb, deduplicated PILE, a subset of RedPajama, and a subset of Dolma v1.6, totaling approximately 1.8 trillion tokens. Please check license agreements and terms of these datasets before using them.
+Our pre-training dataset contains RefinedWeb, deduplicated PILE, a subset of RedPajama, and a subset of Dolma v1.6, totaling approximately 1.8 trillion tokens. 
 
 
 
@@ -28,11 +28,12 @@ Additional arguments to the hugging face generate function can be passed via `ge
 ```
 python generate_openelm.py --model apple/OpenELM-270M --hf_access_token [HF_ACCESS_TOKEN] --prompt 'Once upon a time there was' --generate_kwargs repetition_penalty=1.2 prompt_lookup_num_tokens=10
 ```
-Alternatively, try model-wise speculative generation with an [assistive model](https://huggingface.co/blog/assisted-generation) by passing a smaller model through the `assistant_model` argument, for example:
+Alternatively, model-wise speculative generation with an [assistive model](https://huggingface.co/blog/assisted-generation) can be also tried by passing a smaller model model through the `assistant_model` argument, for example:
 ```
 python generate_openelm.py --model apple/OpenELM-270M --hf_access_token [HF_ACCESS_TOKEN] --prompt 'Once upon a time there was' --generate_kwargs repetition_penalty=1.2 --assistant_model [SMALLER_MODEL]
 ```
 
+
 ## Main Results
 
 ### Zero-Shot
@@ -106,10 +107,9 @@ pip install tokenizers>=0.15.2 transformers>=4.38.2 sentencepiece>=0.2.0
 ```bash
 
 # OpenELM-270M
-hf_model=apple/OpenELM-270M
+hf_model=OpenELM-270M
 
-# this flag is needed because lm-eval-harness set add_bos_token to False by default, but OpenELM uses LLaMA tokenizer which requires add_bos_token to be True
-tokenizer=meta-llama/Llama-2-7b-hf
+# this flag is needed because lm-eval-harness set add_bos_token to False by default, but OpenELM uses LLaMa tokenizer which requires add_bos_token to be True
 add_bos_token=True
 batch_size=1
 
@@ -118,7 +118,7 @@ mkdir lm_eval_output
 shot=0
 task=arc_challenge,arc_easy,boolq,hellaswag,piqa,race,winogrande,sciq,truthfulqa_mc2
 lm_eval --model hf \
-        --model_args pretrained=${hf_model},trust_remote_code=True,add_bos_token=${add_bos_token},tokenizer=${tokenizer} \
+        --model_args pretrained=${hf_model},trust_remote_code=True,add_bos_token=${add_bos_token} \
         --tasks ${task} \
         --device cuda:0 \
         --num_fewshot ${shot} \
@@ -128,7 +128,7 @@ lm_eval --model hf \
 shot=5
 task=mmlu,winogrande
 lm_eval --model hf \
-        --model_args pretrained=${hf_model},trust_remote_code=True,add_bos_token=${add_bos_token},tokenizer=${tokenizer} \
+        --model_args pretrained=${hf_model},trust_remote_code=True,add_bos_token=${add_bos_token} \
         --tasks ${task} \
         --device cuda:0 \
         --num_fewshot ${shot} \
@@ -138,7 +138,7 @@ lm_eval --model hf \
 shot=25
 task=arc_challenge,crows_pairs_english
 lm_eval --model hf \
-        --model_args pretrained=${hf_model},trust_remote_code=True,add_bos_token=${add_bos_token},tokenizer=${tokenizer} \
+        --model_args pretrained=${hf_model},trust_remote_code=True,add_bos_token=${add_bos_token} \
         --tasks ${task} \
         --device cuda:0 \
         --num_fewshot ${shot} \
@@ -148,7 +148,7 @@ lm_eval --model hf \
 shot=10
 task=hellaswag
 lm_eval --model hf \
-        --model_args pretrained=${hf_model},trust_remote_code=True,add_bos_token=${add_bos_token},tokenizer=${tokenizer} \
+        --model_args pretrained=${hf_model},trust_remote_code=True,add_bos_token=${add_bos_token} \
         --tasks ${task} \
         --device cuda:0 \
         --num_fewshot ${shot} \
@@ -160,30 +160,5 @@ lm_eval --model hf \
 
 ## Bias, Risks, and Limitations
 
-The release of OpenELM models aims to empower and enrich the open research community by providing access to state-of-the-art language models. Trained on publicly available datasets, these models are made available without any safety guarantees. Consequently, there exists the possibility of these models producing outputs that are inaccurate, harmful, biased, or objectionable in response to user prompts. Thus, it is imperative for users and developers to undertake thorough safety testing and implement appropriate filtering mechanisms tailored to their specific requirements.
-
-## Citation
-
-If you find our work useful, please cite:
-
-```BibTex 
-@article{mehtaOpenELMEfficientLanguage2024,
-	title = {{OpenELM}: {An} {Efficient} {Language} {Model} {Family} with {Open} {Training} and {Inference} {Framework}},
-	shorttitle = {{OpenELM}},
-	url = {https://arxiv.org/abs/2404.14619v1},
-	language = {en},
-	urldate = {2024-04-24},
-	journal = {arXiv.org},
-	author = {Mehta, Sachin and Sekhavat, Mohammad Hossein and Cao, Qingqing and Horton, Maxwell and Jin, Yanzi and Sun, Chenfan and Mirzadeh, Iman and Najibi, Mahyar and Belenko, Dmitry and Zatloukal, Peter and Rastegari, Mohammad},
-	month = apr,
-	year = {2024},
-}
-
-@inproceedings{mehta2022cvnets, 
-     author = {Mehta, Sachin and Abdolhosseini, Farzad and Rastegari, Mohammad}, 
-     title = {CVNets: High Performance Library for Computer Vision}, 
-     year = {2022}, 
-     booktitle = {Proceedings of the 30th ACM International Conference on Multimedia}, 
-     series = {MM '22} 
-}
-```
+Our OpenELM models are not trained with any safety guarantees, the model outputs can be potentially inaccurate, harmful or contain biased information. produce inaccurate, biased or other objectionable responses to user prompts. Therefore, users and developers should conduct extensive safety testing and filtering suited to their specific needs.
+

modeling_openelm.py

@@ -783,7 +783,7 @@ class OpenELMModel(OpenELMPreTrainedModel):
             )
 
         if self.config._attn_implementation == "sdpa" and attention_mask is not None:
-            # For dynamo, rather use a check on fullgraph=True once this is possible (https://github.com/pytorch/pytorch/pull/120400).
+            # TODO: For dynamo, rather use a check on fullgraph=True once this is possible (https://github.com/pytorch/pytorch/pull/120400).
             is_tracing = (
                 torch.jit.is_tracing()
                 or isinstance(input_tensor, torch.fx.Proxy)
@@ -967,7 +967,7 @@ class OpenELMForCausalLM(OpenELMPreTrainedModel):
             input_ids = input_ids[:, past_length:]
             position_ids = position_ids[:, past_length:]
 
-        # we should only keep a `cache_position` in generate, and do +=1.
+        # TODO @gante we should only keep a `cache_position` in generate, and do +=1.
         # same goes for position ids. Could also help with continued generation.
         cache_position = torch.arange(
             past_length,
@@ -981,7 +981,7 @@ class OpenELMForCausalLM(OpenELMPreTrainedModel):
         else:
             # The `contiguous()` here is necessary to have a static stride during decoding. torchdynamo otherwise
             # recompiles graphs as the stride of the inputs is a guard. Ref: https://github.com/huggingface/transformers/pull/29114
-            # We could use `next_tokens` directly instead.
+            # TODO: use `next_tokens` directly instead.
             model_inputs = {"input_ids": input_ids.contiguous()}
 
         model_inputs.update(