Patent ID: 11868381
Assignee: GOOGLE LLC
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 11:
12. A processing system for training a language model, comprising:
a memory; and
one or more processors coupled to the memory and configured to:
pre-train the language model based on a plurality of pre-training examples each comprising a respective table; and
fine-tune the language model based on a plurality of fine-tuning examples each comprising a respective question, a respective answer, and a corresponding table;
wherein, to fine-tune the language model, the one or more processors are further configured to, for a first fine-tuning example of the plurality of fine-tuning examples having the respective answer being a scalar:
generate an estimated answer to the respective question based on:
the corresponding table;
the language model's predictions of whether the respective answer is based on each cell of a plurality of cells of the corresponding table; and
the language model's predictions of whether the respective answer is based on each aggregation operation of a plurality of aggregation operations;

generate a first loss value based on the estimated answer;
generate a second loss value based on the language model's predictions of whether the respective answer is based on each aggregation operation of the plurality of aggregation operations; and
modify one or more parameters of the language model based at least on the first and second loss values.