title_wip

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/getting-started/example-datasets/tw-weather.md)

This dataset contains historical meteorological observations measurements for the last 128 years. Each row is a measurement for a point in date time and weather station.

The origin of this dataset is available [here](https://github.com/Raingel/historical_weather) and the list of weather station numbers can be found [here](https://github.com/Raingel/weather_station_list).

> The sources of meteorological datasets include the meteorological stations that are established by the Central Weather Administration (station code is beginning with C0, C1, and 4) and the agricultural meteorological stations belonging to the Council of Agriculture (station code other than those mentioned above):

- StationId
- MeasuredDate, the observation time
- StnPres, the station air pressure
- SeaPres, the sea level pressure
- Td, the dew point temperature
- RH, the relative humidity
- Other elements where available

## Downloading the data[​](#downloading-the-data "Direct link to Downloading the data")

- A [pre-processed version](#pre-processed-data) of the data for the ClickHouse, which has been cleaned, re-structured, and enriched. This dataset covers the years from 1896 to 2023.
- [Download the original raw data](#original-raw-data) and convert to the format required by ClickHouse. Users wanting to add their own columns may wish to explore or complete their approaches.

### Pre-processed data[​](#pre-processed-data "Direct link to Pre-processed data")

The dataset has also been re-structured from a measurement per line to a row per weather station id and measured date, i.e.

```
StationId,MeasuredDate,StnPres,Tx,RH,WS,WD,WSGust,WDGust,Precp,GloblRad,TxSoil0cm,TxSoil5cm,TxSoil20cm,TxSoil50cm,TxSoil100cm,SeaPres,Td,PrecpHour,SunShine,TxSoil10cm,EvapA,Visb,UVI,Cloud Amount,TxSoil30cm,TxSoil200cm,TxSoil300cm,TxSoil500cm,VaporPressure
C0X100,2016-01-01 01:00:00,1022.1,16.1,72,1.1,8.0,,,,,,,,,,,,,,,,,,,,,,,
C0X100,2016-01-01 02:00:00,1021.6,16.0,73,1.2,358.0,,,,,,,,,,,,,,,,,,,,,,,
C0X100,2016-01-01 03:00:00,1021.3,15.8,74,1.5,353.0,,,,,,,,,,,,,,,,,,,,,,,
C0X100,2016-01-01 04:00:00,1021.2,15.8,74,1.7,8.0,,,,,,,,,,,,,,,,,,,,,,,
```

It is easy to query and ensure that the resulting table has less sparse and some elements are null because they're not available to be measured in this weather station.

This dataset is available in the following Google CloudStorage location. Either download the dataset to your local filesystem (and insert them with the ClickHouse client) or insert them directly into the ClickHouse (see [Inserting from URL](#inserting-from-url)).

To download:

```
wget https://storage.googleapis.com/taiwan-weather-observaiton-datasets/preprocessed_weather_daily_1896_2023.tar.gz

# Option: Validate the checksum
md5sum preprocessed_weather_daily_1896_2023.tar.gz
# Checksum should be equal to: 11b484f5bd9ddafec5cfb131eb2dd008

tar -xzvf preprocessed_weather_daily_1896_2023.tar.gz
daily_weather_preprocessed_1896_2023.csv

# Option: Validate the checksum
md5sum daily_weather_preprocessed_1896_2023.csv
# Checksum should be equal to: 1132248c78195c43d93f843753881754
```

### Original raw data[​](#original-raw-data "Direct link to Original raw data")

The following details are about the steps to download the original raw data to transform and convert as you want.

#### Download[​](#download "Direct link to Download")

To download the original raw data:

```
mkdir tw_raw_weather_data && cd tw_raw_weather_data

wget https://storage.googleapis.com/taiwan-weather-observaiton-datasets/raw_data_weather_daily_1896_2023.tar.gz

# Option: Validate the checksum
md5sum raw_data_weather_daily_1896_2023.tar.gz
# Checksum should be equal to: b66b9f137217454d655e3004d7d1b51a

tar -xzvf raw_data_weather_daily_1896_2023.tar.gz
466920_1928.csv
466920_1929.csv
466920_1930.csv
466920_1931.csv
...

# Option: Validate the checksum
cat *.csv | md5sum
# Checksum should be equal to: b26db404bf84d4063fac42e576464ce1
```

#### Retrieve the Taiwan weather stations[​](#retrieve-the-taiwan-weather-stations "Direct link to Retrieve the Taiwan weather stations")

```
wget -O weather_sta_list.csv https://github.com/Raingel/weather_station_list/raw/main/data/weather_sta_list.csv

# Option: Convert the UTF-8-BOM to UTF-8 encoding
sed -i '1s/^\xEF\xBB\xBF//' weather_sta_list.csv
```

## Create table schema[​](#create-table-schema "Direct link to Create table schema")

Create the MergeTree table in ClickHouse (from the ClickHouse client).

```
CREATE TABLE tw_weather_data (
    StationId String null,
    MeasuredDate DateTime64,
    StnPres Float64 null,
    SeaPres Float64 null,
    Tx Float64 null,
    Td Float64 null,
    RH Float64 null,
    WS Float64 null,
    WD Float64 null,
    WSGust Float64 null,
    WDGust Float64 null,
    Precp Float64 null,
    PrecpHour Float64 null,
    SunShine Float64 null,
    GloblRad Float64 null,
    TxSoil0cm Float64 null,
    TxSoil5cm Float64 null,
    TxSoil10cm Float64 null,
    TxSoil20cm Float64 null,
    TxSoil50cm Float64 null,
    TxSoil100cm Float64 null,
    TxSoil30cm Float64 null,
    TxSoil200cm Float64 null,
    TxSoil300cm Float64 null,
    TxSoil500cm Float64 null,
    VaporPressure Float64 null,
    UVI Float64 null,
    "Cloud Amount" Float64 null,
    EvapA Float64 null,
    Visb Float64 null
)
ENGINE = MergeTree
ORDER BY (MeasuredDate);
```

## Inserting into ClickHouse[​](#inserting-into-clickhouse "Direct link to Inserting into ClickHouse")

### Inserting from local file[​](#inserting-from-local-file "Direct link to Inserting from local file")

Data can be inserted from a local file as follows (from the ClickHouse client):

```
INSERT INTO tw_weather_data FROM INFILE '/path/to/daily_weather_preprocessed_1896_2023.csv'
```

where `/path/to` represents the specific user path to the local file on the disk.

And the sample response output is as follows after inserting data into the ClickHouse:

```
Query id: 90e4b524-6e14-4855-817c-7e6f98fbeabb

Ok.
131985329 rows in set. Elapsed: 71.770 sec. Processed 131.99 million rows, 10.06 GB (1.84 million rows/s., 140.14 MB/s.)
Peak memory usage: 583.23 MiB.
```

### Inserting from URL[​](#inserting-from-url "Direct link to Inserting from URL")

```
INSERT INTO tw_weather_data SELECT *
FROM url('https://storage.googleapis.com/taiwan-weather-observaiton-datasets/daily_weather_preprocessed_1896_2023.csv', 'CSVWithNames')
```

To know how to speed this up, please see our blog post on [tuning large data loads](https://clickhouse.com/blog/supercharge-your-clickhouse-data-loads-part2).

## Check data rows and sizes[​](#check-data-rows-and-sizes "Direct link to Check data rows and sizes")

1. Let's see how many rows are inserted:

```
SELECT formatReadableQuantity(count())
FROM tw_weather_data;
```

```
┌─formatReadableQuantity(count())─┐
│ 131.99 million                  │
└─────────────────────────────────┘
```

2. Let's see how much disk space are used for this table:

```
SELECT
    formatReadableSize(sum(bytes)) AS disk_size,
    formatReadableSize(sum(data_uncompressed_bytes)) AS uncompressed_size
FROM system.parts
WHERE (`table` = 'tw_weather_data') AND active
```

```
┌─disk_size─┬─uncompressed_size─┐
│ 2.13 GiB  │ 32.94 GiB         │
└───────────┴───────────────────┘
```

## Sample queries[​](#sample-queries "Direct link to Sample queries")

### Q1: Retrieve the highest dew point temperature for each weather station in the specific year[​](#q1-retrieve-the-highest-dew-point-temperature-for-each-weather-station-in-the-specific-year "Direct link to Q1: Retrieve the highest dew point temperature for each weather station in the specific year")

```
SELECT
    StationId,
    max(Td) AS max_td
FROM tw_weather_data
WHERE (year(MeasuredDate) = 2023) AND (Td IS NOT NULL)
GROUP BY StationId