Package: ttqakit
🌐Website|🎥Video|📦PyPI|🤗Huggingface DatasetsTableQAKit: A Toolkit for Table Question Answering🔥 Updates[2023-8-7]: We released ourcode,datasetsandPyPI Package. Check it out!✨ FeaturesTableQAKit is a unified platform for TableQA (especially in the LLM era). Its main features includes:Extensible design: You can use the interfaces defined by the toolkit, extend methods and models, and implement your own new models based on your own data.Equipped with LLM: TableQAKit supports LLM-based methods, including LLM-prompting methods and LLM-finetuning methods.Comprehensive datasets: We design a unified data interface to process data and store them in Huggingface datasets.Powerful methods: Using our toolkit, you can reproduce most of the SOTA methods for TableQA tasks.Efficient LLM benchmark: TableQAEval, a benchmark to evaluate the performance of LLM for TableQA. It evaluates LLM's modeling ability of long tables (context) and comprehension capabilities (numerical reasoning, multi-hop reasoning).Comprehensive Survey: We are about to release a systematic TableQA Survey, this project is a pre-work.⚙️ Installpip install tableqakit
or
git clone git@github.com:lfy79001/TableQAKit.git
pip install -r requirements.txt

pip install ttqakit📁 FolderThe TableQAKit repository is structured as follows:├──icl/# LLM-prompting toolkit├──llama/# LLM-finetuning toolkit├──mmqa_utils/# EncyclopediaQA toolkit│├──classifier_module/# The package for classifier│├──retriever_module/# The package for encyclopedia retrieval├──structuredqa/# Read model TaLMs│├──builder/
│├──utils/
├──retriever/# TableQA's general retriever （SpreadSheet examples）├──multihop/# Readers for encyclopediaQA│├──Retrieval/
│└──Read/
├──numerical/# Readers for some TableQA datasets├──TableQAEval/# The proposed new LLM-Long-Table Benchmark│├──Baselines/# Add your LLMs││├──turbo16k-table.py
││├──llama2-chat-table.py
││└──...
│├──Evaluation/# metrics│└──TableQAEval.json├──outputs/# the results of some models├──loaders/│├──WikiSQL.py
│└──...
├──structs/│├──data.py
├──static/├──LICENSE
└──README.md🗃️ DatasetAccording to our taxonomy, we classify the TableQA task into three categories of tasks, as shown in the following figure:🔧 Get startedRetrieval ModulesQuickStartMultiHiertt Dataset as a demonstrationfrom TableQAKit.retriever import MultiHierttTrainer


trainer = MultiHierttTrainer()# train stage:
trainer.train()# infer stage:
trainer.infer()Trainpython main.py \
--train_mode row \
--per_device_train_batch_size 16 \
--per_device_eval_batch_size 1 \
--dataloader_pin_memory False \
--output_dir ./ckpt \
--train_path ./data/train.json \
--val_path ./data/val.json \
--save_steps 1000 \
--logging_steps 20 \
--learning_rate 0.00001 \
--top_n_for_eval 10 \
--encoder_path ./PLM/bert-base-uncased/Inferencepython infer.py \
--output_dir ./ckpt \
--encoder_path ./ckpt/encoder/deberta-large \
--dataloader_pin_memory False \
--ckpt_for_test ./ckpt/retriever/deberta/epoch1_step30000.pt \
--test_path ./data/MultiHiertt/test.json \
--test_out_path ./prediction.json \
--top_n_for_test 10Create Trainer for New Datasetfrom TableQAKit.retriever import RetrieverTrainer as RT

class NewTrainer(RT):
    def read_data(self, data_path: str) -> List[Dict]:
        """

        :param data_path: The path of data
        :return: List of raw data
        [
            data_1,
            data_2,
            ……
        ]
        """
        data = json.load(
            open(data_path, 'r', encoding='utf-8')
        )
        return data

    def data_proc(self, instance) -> Dict:
        """

        :return:
        {
            "id": str,
            "question": str,
            "rows": list[str],
            "labels": list[int]
        }
        """
        rows = instance["paragraphs"]
        labels = [0] * len(instance["paragraphs"])
        if len(instance["qa"]["text_evidence"]):
            for text_evidence in instance["qa"]["text_evidence"]:
                labels[text_evidence] = 1
        for k, v in instance["table_description"].items():
            rows.append(v)
            labels.append(1 if k in instance["qa"]["table_evidence"] else 0)
        return {
            "id": instance["uid"],
            "question": instance["qa"]["question"],
            "rows": rows,
            "labels": labels
        }LLM-Prompting MethodsCheckhearfor more details.LLM-Finetuning MethodsCheckhearfor more details.Reading ModulesTaLM ReasonerCheckhearfor more details.Multimodal ReasonerCheckhearfor more details.TableQAEvalTableQAEval is a benchmark to evaluate the performance of LLM for TableQA. It evaluates LLM's modeling ability of long tables (context) and comprehension capabilities (numerical reasoning, multi-hop reasoning).LeaderboardModelParametersNumerical ReasoningMulti-hop ReasoningStructured ReasoningTotalTurbo-16k-0613-20.352.854.343.5LLaMA2-7b-chat7B2.014.213.412.6ChatGLM2-6b-8k6B1.410.111.510.2LLaMA2-7b-4k7B0.89.25.46.6longchat-7b-16k7B0.37.15.15.2LLaMA-7b-2k7B0.57.34.14.5MPT-7b-65k7B0.33.22.02.3LongLLaMA-3b3B0.04.31.72.0More details are shown inTableQAEval.✅ TODOWe will continue to optimize the toolkit.AcknowledgePrimary contributors: Fangyu Lei, Tongxu Luo, Pengqi Yang, Weihao Liu, Hanwen Liu, Jiahe Lei, Yifan Wei, Shizhu He and Kang Liu.Thank you very much to Yilun Zhao（Yale University）and Yongwei Zhou (HIT) for their assistance.