Title: Support Apache TsFile Datasets
State: open
Labels: description: New feature or request
name: enhancement

### Feature request

I would love to use Hugging Face datasets library to directly load datasets composed of .tsfile files, for example:

`ds = load_dataset("username/dataset-with-tsfile-files")`

This feature would allow researchers working on time-series tasks to seamlessly integrate datasets stored in the Apache TsFile format into the Hugging Face ecosystem.

### Motivation

[Apache TsFile](https://tsfile.apache.org/) is a mature Apache project and a dedicated file format designed for efficient time-series data storage and retrieval. The repository is [here](https://github.com/apache/tsfile).

It has been widely adopted in the IoT community and serves as the underlying storage format for projects like [Apache IoTDB](https://iotdb.apache.org/).

Apache TsFile has the following advantages in the time-series area:
- Time-series native schema. Time-series data is organized by device and sensor IDs.
- A complete multi-language API (Python, Java, C++, C) for reading and writing tsfile.
- Superior write throughput and query efficiency.
- High compression ratio through per-series encoding and compression schemes.
- Efficient dataset transformation. ETL-free file compaction and efficient random access to time-series chunks, enabling faster data loading and lower query latency.

These properties make TsFile highly suitable for time-series model training, especially where time-series random access and efficient I/O are critical.

More details can be referred from this paper “[Apache TsFile: An IoT-native Time Series File Format (VLDB 2024)](https://www.vldb.org/pvldb/vol17/p4064-song.pdf)”.

Integrating TsFile support into datasets will benefit the broader machine learning community working on tasks such as forecasting and anomaly detection.

### Your contribution

As a member of the TsFile community, I recently initiated a [proposal ](https://lists.apache.org/thread/119vc9nh03dz4583cx9fwt83fp8v68vy)to integrate TsFile with Huggingface, which has received enthusiastic responses from the community. 

We are willing to do the following contributions:
- Implement and contribute the PR that adds TsFile dataset support to Hugging Face datasets.
- Provide long-term maintenance for this integration.
- Any other needs for TsFile to support large-scale time-series datasets.

We are excited to contribute and continuously participate in the future evolution of TsFile and datasets to better support time-series data workload.

Comments:
A large quantity of industrial timeseries data has been stored as TsFile, and I have been constantly hearing about AI fellows complaining about the lack of data or the insufficiency of data quality.

I like the ambition that uses TsFile as the bridge between AI research and industrial analysis requirements. This may help both sides improve their works with high-quality data and realtime data access.

It will be so convenient to have such a method to directly load tsfile into memory for further analysis.

Looking forward to see the tsfile become the part of the AI eco-systems.

Looking forward to the support for TsFile format!

Hey folks! I’ve added TsFile support by following the existing HDF5/Parquet patterns.

This includes:

A TsFile builder with schema inference from file metadata

Time-range filtering and column selection

Memory-efficient reading using the tsfile library’s iterator API

11 tests, all passing ✅

I’ll be opening a PR shortly, would love any suggestions or feedback you might have!