The fine-tuning approach used in this work has several limitations. Primarily, it relies on a single type of input representation (e.g., SMILES strings) and optimizes solely for generative loss, which may not fully capture the nuances of regression tasks. This could lead to suboptimal performance, especially when compared to models that utilize more detailed structural information or are trained with loss functions specifically designed for regression, such as mean squared error.
Alternative training strategies, such as prompt engineering, could significantly enhance the regression capabilities of LLMs. By carefully designing prompts that provide context or specify the desired output format, the model can be guided to produce more accurate predictions. For instance, including examples of input-output pairs in the prompt could help the model understand the relationship between molecular structures and their properties better.
Multitask learning is another promising strategy. By training the LLM on multiple related tasks simultaneously, the model can learn shared representations that improve its generalization capabilities. For example, if the model is trained to predict various molecular properties (e.g., formation energy, HOMO, LUMO) at the same time, it may develop a more comprehensive understanding of the underlying chemical principles, leading to improved performance on individual tasks.