Repository: ecmwf/pdbufr (language: python)

Issue:
Is it possible to automatically take every key from bufr_dump as an argument for columns in bufr_read?
Is it possible to automate the opening of a bufr file without setting the specific columns? So instead of explicitly setting the "columns" argument, pdbufr would read all of the contents with each key in bufr_dump as a pandas column?

Thanks for any feedback!

Is it possible to automatically take every key from bufr_dump as an argument for columns in bufr_read?
Is it possible to automate the opening of a bufr file without setting the specific columns? So instead of explicitly setting the "columns" argument, pdbufr would read all of the contents with each key in bufr_dump as a pandas column?

Thanks for any feedback!



Discussion / hints:
Hi @ShaneMill1,
Thank you for your query. Unfortunately, this is not possible in pdbufr, but we agree it would be a useful feature.  We have a lot of high priority work at the moment and we plan to look into this in the second half of the year.

Kind regards,
Sandor
Hey @sandorkertesz thank you for letting us know!
Hi @ShaneMill1,
Thank you for your query. Unfortunately, this is not possible in pdbufr, but we agree it would be a useful feature.  We have a lot of high priority work at the moment and we plan to look into this in the second half of the year.

Kind regards,
Sandor
Hey @sandorkertesz thank you for letting us know!

Fix patch:
diff --git a/.github/workflows/on-push.yml b/.github/workflows/on-push.yml
index 73bc59d..f8eb469 100644
--- a/.github/workflows/on-push.yml
+++ b/.github/workflows/on-push.yml
@@ -60,6 +60,7 @@ jobs:
         conda install ipykernel
         conda install pandoc
         conda install nbsphinx
+        conda install jinja2=3.0.3
         pip install --no-deps -e .
         python setup.py build_sphinx
     # - name: Test README with pytest
diff --git a/MANIFEST.in b/MANIFEST.in
index a96e913..b372f46 100644
--- a/MANIFEST.in
+++ b/MANIFEST.in
@@ -7,5 +7,6 @@ include mypy.ini
 recursive-include tests *.bufr
 recursive-include tests *.py
 recursive-include tests *.yml
+recursive-include tests *.csv
 recursive-include docs *
 
diff --git a/docs/_static/flat_dump_output.txt b/docs/_static/flat_dump_output.txt
new file mode 100644
index 0000000..aad2003
--- /dev/null
+++ b/docs/_static/flat_dump_output.txt
@@ -0,0 +1,5 @@
+    subsetNumber  #1#blockNumber  #1#stationNumber    ...  #25#airTemperature  #25#dewpointTemperature  #25#windDirection  #25#windSpeed
+0        1             71               907                    NaN                      NaN                NaN            NaN
+1        1             71               823                  221.5                    191.5                NaN            NaN
+
+[2 rows x 197 columns]
\ No newline at end of file
diff --git a/docs/_static/h_dump_output.txt b/docs/_static/h_dump_output.txt
new file mode 100644
index 0000000..83ee73f
--- /dev/null
+++ b/docs/_static/h_dump_output.txt
@@ -0,0 +1,10 @@
+    latitude  longitude  pressure  airTemperature
+0      58.47     -78.08  100300.0           258.3
+1      58.47     -78.08  100000.0           259.7
+2      58.47     -78.08   99800.0           261.1
+...
+46     53.75     -73.67   25000.0           221.1
+47     53.75     -73.67   23200.0           223.1
+48     53.75     -73.67   20500.0           221.5
+
+[48 rows x 4 columns]
\ No newline at end of file
diff --git a/docs/_static/temp_structure.png b/docs/_static/temp_structure.png
new file mode 100644
index 0000000..68d19b5
Binary files /dev/null and b/docs/_static/temp_structure.png differ
diff --git a/docs/examples.rst b/docs/examples.rst
index 03c6292..f77d0dd 100644
--- a/docs/examples.rst
+++ b/docs/examples.rst
@@ -6,7 +6,14 @@ Examples
 Here is a list of example notebooks to illustrate how to extract BUFR data using pdbufr.
 
 .. toctree::
-    :maxdepth: 2
+    :maxdepth: 1
     :glob:
 
-    examples/*
+    examples/aircraft
+    examples/ens
+    examples/radiosonde
+    examples/sat
+    examples/synop
+    examples/tropical_cyclone
+    examples/flat_dump
+
diff --git a/docs/examples/flat_dump.ipynb b/docs/examples/flat_dump.ipynb
new file mode 100644
index 0000000..ee0b336
--- /dev/null
+++ b/docs/examples/flat_dump.ipynb
@@ -0,0 +1,1665 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!test -f aircraft_small.bufr || wget https://github.com/ecmwf/pdbufr/raw/master/tests/sample-data/aircraft_small.bufr\n",
+    "!test -f temp.bufr || wget https://github.com/ecmwf/pdbufr/raw/master/tests/sample-data/temp.bufr"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Flat dump"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pdbufr"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The flat dump mode is activated when [read_bufr()](../read_bufr.rst) is called with ``flat`` = ``True``. In this mode messages/subsets are extracted as a whole preserving the column order (see exceptions below). \n",
+    "\n",
+    "Since the results contain a large number of columns with very long names the **transpose** of the DataFrames are shown in all the examples below to make better use of the available space."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Options\n",
+    "\n",
+    "By default all the header and data keys are extracted:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>0</th>\n",
+       "      <th>1</th>\n",
+       "      <th>2</th>\n",
+       "      <th>3</th>\n",
+       "      <th>4</th>\n",
+       "      <th>5</th>\n",
+       "      <th>6</th>\n",
+       "      <th>7</th>\n",
+       "      <th>8</th>\n",
+       "      <th>9</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>edition</th>\n",
+       "      <td>3</td>\n",
+       "      <td>3</td>\n",
+       "      <td>3</td>\n",
+       "      <td>3</td>\n",
+       "      <td>3</td>\n",
+       "      <td>3</td>\n",
+       "      <td>3</td>\n",
+       "      <td>3</td>\n",
+       "      <td>3</td>\n",
+       "      <td>3</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>masterTableNumber</th>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>bufrHeaderSubCentre</th>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>bufrHeaderCentre</th>\n",
+       "      <td>98</td>\n",
+       "      <td>98</td>\n",
+       "      <td>98</td>\n",
+       "      <td>98</td>\n",
+       "      <td>98</td>\n",
+       "      <td>98</td>\n",
+       "      <td>98</td>\n",
+       "      <td>98</td>\n",
+       "      <td>98</td>\n",
+       "      <td>98</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>updateSequenceNumber</th>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>...</th>\n",
+       "      <td>...</td>\n",
+       "      <td>...</td>\n",
+       "      <td>...</td>\n",
+       "      <td>...</td>\n",
+       "      <td>...</td>\n",
+       "      <td>...</td>\n",
+       "      <td>...</td>\n",
+       "      <td>...</td>\n",
+       "      <td>...</td>\n",
+       "      <td>...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#dewpointTemperature</th>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#relativeHumidity</th>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#airframeIcing</th>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#centre</th>\n",
+       "      <td>98</td>\n",
+       "      <td>98</td>\n",
+       "      <td>98</td>\n",
+       "      <td>98</td>\n",
+       "      <td>98</td>\n",
+       "      <td>98</td>\n",
+       "      <td>98</td>\n",
+       "      <td>98</td>\n",
+       "      <td>98</td>\n",
+       "      <td>98</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#generatingApplication</th>\n",
+       "      <td>1</td>\n",
+       "      <td>1</td>\n",
+       "      <td>1</td>\n",
+       "      <td>1</td>\n",
+       "      <td>1</td>\n",
+       "      <td>1</td>\n",
+       "      <td>1</td>\n",
+       "      <td>1</td>\n",
+       "      <td>1</td>\n",
+       "      <td>1</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "<p>81 rows × 10 columns</p>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "                             0     1     2     3     4     5     6     7  \\\n",
+       "edition                      3     3     3     3     3     3     3     3   \n",
+       "masterTableNumber            0     0     0     0     0     0     0     0   \n",
+       "bufrHeaderSubCentre          0     0     0     0     0     0     0     0   \n",
+       "bufrHeaderCentre            98    98    98    98    98    98    98    98   \n",
+       "updateSequenceNumber         0     0     0     0     0     0     0     0   \n",
+       "...                        ...   ...   ...   ...   ...   ...   ...   ...   \n",
+       "#1#dewpointTemperature    None  None  None  None  None  None  None  None   \n",
+       "#1#relativeHumidity       None  None  None  None  None  None  None  None   \n",
+       "#1#airframeIcing          None  None  None  None  None  None  None  None   \n",
+       "#1#centre                   98    98    98    98    98    98    98    98   \n",
+       "#1#generatingApplication     1     1     1     1     1     1     1     1   \n",
+       "\n",
+       "                             8     9  \n",
+       "edition                      3     3  \n",
+       "masterTableNumber            0     0  \n",
+       "bufrHeaderSubCentre          0     0  \n",
+       "bufrHeaderCentre            98    98  \n",
+       "updateSequenceNumber         0     0  \n",
+       "...                        ...   ...  \n",
+       "#1#dewpointTemperature    None  None  \n",
+       "#1#relativeHumidity       None  None  \n",
+       "#1#airframeIcing          None  None  \n",
+       "#1#centre                   98    98  \n",
+       "#1#generatingApplication     1     1  \n",
+       "\n",
+       "[81 rows x 10 columns]"
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "df = pdbufr.read_bufr(\"aircraft_small.bufr\", flat=True)\n",
+    "df.T"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "However, we can extract only the **header keys**:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>0</th>\n",
+       "      <th>1</th>\n",
+       "      <th>2</th>\n",
+       "      <th>3</th>\n",
+       "      <th>4</th>\n",
+       "      <th>5</th>\n",
+       "      <th>6</th>\n",
+       "      <th>7</th>\n",
+       "      <th>8</th>\n",
+       "      <th>9</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>edition</th>\n",
+       "      <td>3</td>\n",
+       "      <td>3</td>\n",
+       "      <td>3</td>\n",
+       "      <td>3</td>\n",
+       "      <td>3</td>\n",
+       "      <td>3</td>\n",
+       "      <td>3</td>\n",
+       "      <td>3</td>\n",
+       "      <td>3</td>\n",
+       "      <td>3</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>masterTableNumber</th>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>bufrHeaderSubCentre</th>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>bufrHeaderCentre</th>\n",
+       "      <td>98</td>\n",
+       "      <td>98</td>\n",
+       "      <td>98</td>\n",
+       "      <td>98</td>\n",
+       "      <td>98</td>\n",
+       "      <td>98</td>\n",
+       "      <td>98</td>\n",
+       "      <td>98</td>\n",
+       "      <td>98</td>\n",
+       "      <td>98</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>updateSequenceNumber</th>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>dataCategory</th>\n",
+       "      <td>4</td>\n",
+       "      <td>4</td>\n",
+       "      <td>4</td>\n",
+       "      <td>4</td>\n",
+       "      <td>4</td>\n",
+       "      <td>4</td>\n",
+       "      <td>4</td>\n",
+       "      <td>4</td>\n",
+       "      <td>4</td>\n",
+       "      <td>4</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "                       0   1   2   3   4   5   6   7   8   9\n",
+       "edition                3   3   3   3   3   3   3   3   3   3\n",
+       "masterTableNumber      0   0   0   0   0   0   0   0   0   0\n",
+       "bufrHeaderSubCentre    0   0   0   0   0   0   0   0   0   0\n",
+       "bufrHeaderCentre      98  98  98  98  98  98  98  98  98  98\n",
+       "updateSequenceNumber   0   0   0   0   0   0   0   0   0   0\n",
+       "dataCategory           4   4   4   4   4   4   4   4   4   4"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "df = pdbufr.read_bufr(\"aircraft_small.bufr\", columns=\"header\", flat=True)\n",
+    "df.T[:6]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "or only the **data keys**:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>0</th>\n",
+       "      <th>1</th>\n",
+       "      <th>2</th>\n",
+       "      <th>3</th>\n",
+       "      <th>4</th>\n",
+       "      <th>5</th>\n",
+       "      <th>6</th>\n",
+       "      <th>7</th>\n",
+       "      <th>8</th>\n",
+       "      <th>9</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>subsetNumber</th>\n",
+       "      <td>1</td>\n",
+       "      <td>1</td>\n",
+       "      <td>1</td>\n",
+       "      <td>1</td>\n",
+       "      <td>1</td>\n",
+       "      <td>1</td>\n",
+       "      <td>1</td>\n",
+       "      <td>1</td>\n",
+       "      <td>1</td>\n",
+       "      <td>1</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#aircraftFlightNumber</th>\n",
+       "      <td>QGOBTRRA</td>\n",
+       "      <td>QGOBTRRA</td>\n",
+       "      <td>UOZDOZ2S</td>\n",
+       "      <td>UOZDOZ2S</td>\n",
+       "      <td>UOZDOZ2S</td>\n",
+       "      <td>UOZDOZ2S</td>\n",
+       "      <td>VUVTEWZQ</td>\n",
+       "      <td>4IPASOZA</td>\n",
+       "      <td>WSSASKBA</td>\n",
+       "      <td>WSSASKBA</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#aircraftRegistrationNumberOrOtherIdentification</th>\n",
+       "      <td>HGSKJFBA</td>\n",
+       "      <td>HGSKJFBA</td>\n",
+       "      <td>O2RYR4JA</td>\n",
+       "      <td>O2RYR4JA</td>\n",
+       "      <td>O2RYR4JA</td>\n",
+       "      <td>O2RYR4JA</td>\n",
+       "      <td>4NK13QZA</td>\n",
+       "      <td>0IKWU1JA</td>\n",
+       "      <td>P4MAWDZA</td>\n",
+       "      <td>P4MAWDZA</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#aircraftNavigationalSystem</th>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#aircraftDataRelaySystemType</th>\n",
+       "      <td>3</td>\n",
+       "      <td>3</td>\n",
+       "      <td>3</td>\n",
+       "      <td>3</td>\n",
+       "      <td>3</td>\n",
+       "      <td>3</td>\n",
+       "      <td>3</td>\n",
+       "      <td>3</td>\n",
+       "      <td>3</td>\n",
+       "      <td>3</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#instrumentationForWindMeasurement</th>\n",
+       "      <td>4</td>\n",
+       "      <td>4</td>\n",
+       "      <td>4</td>\n",
+       "      <td>4</td>\n",
+       "      <td>4</td>\n",
+       "      <td>4</td>\n",
+       "      <td>4</td>\n",
+       "      <td>4</td>\n",
+       "      <td>4</td>\n",
+       "      <td>4</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#temperatureObservationPrecision</th>\n",
+       "      <td>0.1</td>\n",
+       "      <td>0.1</td>\n",
+       "      <td>0.25</td>\n",
+       "      <td>0.25</td>\n",
+       "      <td>0.25</td>\n",
+       "      <td>0.25</td>\n",
+       "      <td>0.25</td>\n",
+       "      <td>0.25</td>\n",
+       "      <td>0.25</td>\n",
+       "      <td>0.25</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#originalSpecificationOfLatitudeOrLongitude</th>\n",
+       "      <td>1</td>\n",
+       "      <td>1</td>\n",
+       "      <td>10</td>\n",
+       "      <td>10</td>\n",
+       "      <td>10</td>\n",
+       "      <td>10</td>\n",
+       "      <td>10</td>\n",
+       "      <td>1</td>\n",
+       "      <td>10</td>\n",
+       "      <td>10</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#aircraftRollAngle</th>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#stationType</th>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#year</th>\n",
+       "      <td>2009</td>\n",
+       "      <td>2009</td>\n",
+       "      <td>2009</td>\n",
+       "      <td>2009</td>\n",
+       "      <td>2009</td>\n",
+       "      <td>2009</td>\n",
+       "      <td>2009</td>\n",
+       "      <td>2009</td>\n",
+       "      <td>2009</td>\n",
+       "      <td>2009</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#month</th>\n",
+       "      <td>1</td>\n",
+       "      <td>1</td>\n",
+       "      <td>1</td>\n",
+       "      <td>1</td>\n",
+       "      <td>1</td>\n",
+       "      <td>1</td>\n",
+       "      <td>1</td>\n",
+       "      <td>1</td>\n",
+       "      <td>1</td>\n",
+       "      <td>1</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#day</th>\n",
+       "      <td>23</td>\n",
+       "      <td>23</td>\n",
+       "      <td>23</td>\n",
+       "      <td>23</td>\n",
+       "      <td>23</td>\n",
+       "      <td>23</td>\n",
+       "      <td>23</td>\n",
+       "      <td>23</td>\n",
+       "      <td>23</td>\n",
+       "      <td>23</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#hour</th>\n",
+       "      <td>13</td>\n",
+       "      <td>13</td>\n",
+       "      <td>12</td>\n",
+       "      <td>12</td>\n",
+       "      <td>13</td>\n",
+       "      <td>13</td>\n",
+       "      <td>13</td>\n",
+       "      <td>13</td>\n",
+       "      <td>13</td>\n",
+       "      <td>13</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#minute</th>\n",
+       "      <td>0</td>\n",
+       "      <td>1</td>\n",
+       "      <td>56</td>\n",
+       "      <td>58</td>\n",
+       "      <td>0</td>\n",
+       "      <td>2</td>\n",
+       "      <td>2</td>\n",
+       "      <td>2</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#latitude</th>\n",
+       "      <td>35.1</td>\n",
+       "      <td>35.07</td>\n",
+       "      <td>41.45</td>\n",
+       "      <td>41.52</td>\n",
+       "      <td>41.59</td>\n",
+       "      <td>41.67</td>\n",
+       "      <td>38.26</td>\n",
+       "      <td>19.62</td>\n",
+       "      <td>42.31</td>\n",
+       "      <td>42.29</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#longitude</th>\n",
+       "      <td>-89.97</td>\n",
+       "      <td>-89.97</td>\n",
+       "      <td>-75.43</td>\n",
+       "      <td>-75.63</td>\n",
+       "      <td>-75.87</td>\n",
+       "      <td>-76.16</td>\n",
+       "      <td>-78.57</td>\n",
+       "      <td>73.75</td>\n",
+       "      <td>-70.7</td>\n",
+       "      <td>-70.67</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#phaseOfAircraftFlight</th>\n",
+       "      <td>6.0</td>\n",
+       "      <td>6.0</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>3.0</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>5.0</td>\n",
+       "      <td>5.0</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "                                                           0         1  \\\n",
+       "subsetNumber                                               1         1   \n",
+       "#1#aircraftFlightNumber                             QGOBTRRA  QGOBTRRA   \n",
+       "#1#aircraftRegistrationNumberOrOtherIdentification  HGSKJFBA  HGSKJFBA   \n",
+       "#1#aircraftNavigationalSystem                           None      None   \n",
+       "#1#aircraftDataRelaySystemType                             3         3   \n",
+       "#1#instrumentationForWindMeasurement                       4         4   \n",
+       "#1#temperatureObservationPrecision                       0.1       0.1   \n",
+       "#1#originalSpecificationOfLatitudeOrLongitude              1         1   \n",
+       "#1#aircraftRollAngle                                    None      None   \n",
+       "#1#stationType                                             0         0   \n",
+       "#1#year                                                 2009      2009   \n",
+       "#1#month                                                   1         1   \n",
+       "#1#day                                                    23        23   \n",
+       "#1#hour                                                   13        13   \n",
+       "#1#minute                                                  0         1   \n",
+       "#1#latitude                                             35.1     35.07   \n",
+       "#1#longitude                                          -89.97    -89.97   \n",
+       "#1#phaseOfAircraftFlight                                 6.0       6.0   \n",
+       "\n",
+       "                                                           2         3  \\\n",
+       "subsetNumber                                               1         1   \n",
+       "#1#aircraftFlightNumber                             UOZDOZ2S  UOZDOZ2S   \n",
+       "#1#aircraftRegistrationNumberOrOtherIdentification  O2RYR4JA  O2RYR4JA   \n",
+       "#1#aircraftNavigationalSystem                           None      None   \n",
+       "#1#aircraftDataRelaySystemType                             3         3   \n",
+       "#1#instrumentationForWindMeasurement                       4         4   \n",
+       "#1#temperatureObservationPrecision                      0.25      0.25   \n",
+       "#1#originalSpecificationOfLatitudeOrLongitude             10        10   \n",
+       "#1#aircraftRollAngle                                    None      None   \n",
+       "#1#stationType                                             0         0   \n",
+       "#1#year                                                 2009      2009   \n",
+       "#1#month                                                   1         1   \n",
+       "#1#day                                                    23        23   \n",
+       "#1#hour                                                   12        12   \n",
+       "#1#minute                                                 56        58   \n",
+       "#1#latitude                                            41.45     41.52   \n",
+       "#1#longitude                                          -75.43    -75.63   \n",
+       "#1#phaseOfAircraftFlight                                 NaN       NaN   \n",
+       "\n",
+       "                                                           4         5  \\\n",
+       "subsetNumber                                               1         1   \n",
+       "#1#aircraftFlightNumber                             UOZDOZ2S  UOZDOZ2S   \n",
+       "#1#aircraftRegistrationNumberOrOtherIdentification  O2RYR4JA  O2RYR4JA   \n",
+       "#1#aircraftNavigationalSystem                           None      None   \n",
+       "#1#aircraftDataRelaySystemType                             3         3   \n",
+       "#1#instrumentationForWindMeasurement                       4         4   \n",
+       "#1#temperatureObservationPrecision                      0.25      0.25   \n",
+       "#1#originalSpecificationOfLatitudeOrLongitude             10        10   \n",
+       "#1#aircraftRollAngle                                    None      None   \n",
+       "#1#stationType                                             0         0   \n",
+       "#1#year                                                 2009      2009   \n",
+       "#1#month                                                   1         1   \n",
+       "#1#day                                                    23        23   \n",
+       "#1#hour                                                   13        13   \n",
+       "#1#minute                                                  0         2   \n",
+       "#1#latitude                                            41.59     41.67   \n",
+       "#1#longitude                                          -75.87    -76.16   \n",
+       "#1#phaseOfAircraftFlight                                 NaN       NaN   \n",
+       "\n",
+       "                                                           6         7  \\\n",
+       "subsetNumber                                               1         1   \n",
+       "#1#aircraftFlightNumber                             VUVTEWZQ  4IPASOZA   \n",
+       "#1#aircraftRegistrationNumberOrOtherIdentification  4NK13QZA  0IKWU1JA   \n",
+       "#1#aircraftNavigationalSystem                           None      None   \n",
+       "#1#aircraftDataRelaySystemType                             3         3   \n",
+       "#1#instrumentationForWindMeasurement                       4         4   \n",
+       "#1#temperatureObservationPrecision                      0.25      0.25   \n",
+       "#1#originalSpecificationOfLatitudeOrLongitude             10         1   \n",
+       "#1#aircraftRollAngle                                    None      None   \n",
+       "#1#stationType                                             0         0   \n",
+       "#1#year                                                 2009      2009   \n",
+       "#1#month                                                   1         1   \n",
+       "#1#day                                                    23        23   \n",
+       "#1#hour                                                   13        13   \n",
+       "#1#minute                                                  2         2   \n",
+       "#1#latitude                                            38.26     19.62   \n",
+       "#1#longitude                                          -78.57     73.75   \n",
+       "#1#phaseOfAircraftFlight                                 3.0       NaN   \n",
+       "\n",
+       "                                                           8         9  \n",
+       "subsetNumber                                               1         1  \n",
+       "#1#aircraftFlightNumber                             WSSASKBA  WSSASKBA  \n",
+       "#1#aircraftRegistrationNumberOrOtherIdentification  P4MAWDZA  P4MAWDZA  \n",
+       "#1#aircraftNavigationalSystem                           None      None  \n",
+       "#1#aircraftDataRelaySystemType                             3         3  \n",
+       "#1#instrumentationForWindMeasurement                       4         4  \n",
+       "#1#temperatureObservationPrecision                      0.25      0.25  \n",
+       "#1#originalSpecificationOfLatitudeOrLongitude             10        10  \n",
+       "#1#aircraftRollAngle                                    None      None  \n",
+       "#1#stationType                                             0         0  \n",
+       "#1#year                                                 2009      2009  \n",
+       "#1#month                                                   1         1  \n",
+       "#1#day                                                    23        23  \n",
+       "#1#hour                                                   13        13  \n",
+       "#1#minute                                                  0         0  \n",
+       "#1#latitude                                            42.31     42.29  \n",
+       "#1#longitude                                           -70.7    -70.67  \n",
+       "#1#phaseOfAircraftFlight                                 5.0       5.0  "
+      ]
+     },
+     "execution_count": 5,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "df = pdbufr.read_bufr(\"aircraft_small.bufr\", columns=\"data\", flat=True)\n",
+    "df.T[:18]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Filtering works similarly to the hierarchical (i.e. non-flat) mode:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>0</th>\n",
+       "      <th>1</th>\n",
+       "      <th>2</th>\n",
+       "      <th>3</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>subsetNumber</th>\n",
+       "      <td>1</td>\n",
+       "      <td>1</td>\n",
+       "      <td>1</td>\n",
+       "      <td>1</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#aircraftFlightNumber</th>\n",
+       "      <td>UOZDOZ2S</td>\n",
+       "      <td>UOZDOZ2S</td>\n",
+       "      <td>UOZDOZ2S</td>\n",
+       "      <td>UOZDOZ2S</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#aircraftRegistrationNumberOrOtherIdentification</th>\n",
+       "      <td>O2RYR4JA</td>\n",
+       "      <td>O2RYR4JA</td>\n",
+       "      <td>O2RYR4JA</td>\n",
+       "      <td>O2RYR4JA</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#aircraftNavigationalSystem</th>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#aircraftDataRelaySystemType</th>\n",
+       "      <td>3</td>\n",
+       "      <td>3</td>\n",
+       "      <td>3</td>\n",
+       "      <td>3</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#instrumentationForWindMeasurement</th>\n",
+       "      <td>4</td>\n",
+       "      <td>4</td>\n",
+       "      <td>4</td>\n",
+       "      <td>4</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#temperatureObservationPrecision</th>\n",
+       "      <td>0.25</td>\n",
+       "      <td>0.25</td>\n",
+       "      <td>0.25</td>\n",
+       "      <td>0.25</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#originalSpecificationOfLatitudeOrLongitude</th>\n",
+       "      <td>10</td>\n",
+       "      <td>10</td>\n",
+       "      <td>10</td>\n",
+       "      <td>10</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#aircraftRollAngle</th>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#stationType</th>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "      <td>0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#year</th>\n",
+       "      <td>2009</td>\n",
+       "      <td>2009</td>\n",
+       "      <td>2009</td>\n",
+       "      <td>2009</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#month</th>\n",
+       "      <td>1</td>\n",
+       "      <td>1</td>\n",
+       "      <td>1</td>\n",
+       "      <td>1</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#day</th>\n",
+       "      <td>23</td>\n",
+       "      <td>23</td>\n",
+       "      <td>23</td>\n",
+       "      <td>23</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#hour</th>\n",
+       "      <td>12</td>\n",
+       "      <td>12</td>\n",
+       "      <td>13</td>\n",
+       "      <td>13</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#minute</th>\n",
+       "      <td>56</td>\n",
+       "      <td>58</td>\n",
+       "      <td>0</td>\n",
+       "      <td>2</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#latitude</th>\n",
+       "      <td>41.45</td>\n",
+       "      <td>41.52</td>\n",
+       "      <td>41.59</td>\n",
+       "      <td>41.67</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#longitude</th>\n",
+       "      <td>-75.43</td>\n",
+       "      <td>-75.63</td>\n",
+       "      <td>-75.87</td>\n",
+       "      <td>-76.16</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#phaseOfAircraftFlight</th>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "                                                           0         1  \\\n",
+       "subsetNumber                                               1         1   \n",
+       "#1#aircraftFlightNumber                             UOZDOZ2S  UOZDOZ2S   \n",
+       "#1#aircraftRegistrationNumberOrOtherIdentification  O2RYR4JA  O2RYR4JA   \n",
+       "#1#aircraftNavigationalSystem                           None      None   \n",
+       "#1#aircraftDataRelaySystemType                             3         3   \n",
+       "#1#instrumentationForWindMeasurement                       4         4   \n",
+       "#1#temperatureObservationPrecision                      0.25      0.25   \n",
+       "#1#originalSpecificationOfLatitudeOrLongitude             10        10   \n",
+       "#1#aircraftRollAngle                                    None      None   \n",
+       "#1#stationType                                             0         0   \n",
+       "#1#year                                                 2009      2009   \n",
+       "#1#month                                                   1         1   \n",
+       "#1#day                                                    23        23   \n",
+       "#1#hour                                                   12        12   \n",
+       "#1#minute                                                 56        58   \n",
+       "#1#latitude                                            41.45     41.52   \n",
+       "#1#longitude                                          -75.43    -75.63   \n",
+       "#1#phaseOfAircraftFlight                                None      None   \n",
+       "\n",
+       "                                                           2         3  \n",
+       "subsetNumber                                               1         1  \n",
+       "#1#aircraftFlightNumber                             UOZDOZ2S  UOZDOZ2S  \n",
+       "#1#aircraftRegistrationNumberOrOtherIdentification  O2RYR4JA  O2RYR4JA  \n",
+       "#1#aircraftNavigationalSystem                           None      None  \n",
+       "#1#aircraftDataRelaySystemType                             3         3  \n",
+       "#1#instrumentationForWindMeasurement                       4         4  \n",
+       "#1#temperatureObservationPrecision                      0.25      0.25  \n",
+       "#1#originalSpecificationOfLatitudeOrLongitude             10        10  \n",
+       "#1#aircraftRollAngle                                    None      None  \n",
+       "#1#stationType                                             0         0  \n",
+       "#1#year                                                 2009      2009  \n",
+       "#1#month                                                   1         1  \n",
+       "#1#day                                                    23        23  \n",
+       "#1#hour                                                   13        13  \n",
+       "#1#minute                                                  0         2  \n",
+       "#1#latitude                                            41.59     41.67  \n",
+       "#1#longitude                                          -75.87    -76.16  \n",
+       "#1#phaseOfAircraftFlight                                None      None  "
+      ]
+     },
+     "execution_count": 6,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "df = pdbufr.read_bufr(\"aircraft_small.bufr\", \n",
+    "    columns=\"data\", \n",
+    "    filters={\"aircraftFlightNumber\": \"UOZDOZ2S\"}, \n",
+    "    flat=True)\n",
+    "df.T.iloc[:18]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Column alignment\n",
+    "\n",
+    "The aircraft messages we have examined so far had identical structure; each message contained the very same keys in the very same order. The result was always a nicely aligned DataFrame.\n",
+    "\n",
+    "However, in a BUFR file each message can have a different structure and the alignment is not guaranteed at all. We will demonstrate it with a BUFR file containing radiosonde data.\n",
+    "\n",
+    "First, we extract the first message only. From the output we can see it contains 24 pressure level blocks."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>0</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>#23#pressure</th>\n",
+       "      <td>26300.0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#23#verticalSoundingSignificance</th>\n",
+       "      <td>4</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#23#nonCoordinateGeopotential</th>\n",
+       "      <td>89290.0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#23#airTemperature</th>\n",
+       "      <td>218.5</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#23#dewpointTemperature</th>\n",
+       "      <td>198.5</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#23#windDirection</th>\n",
+       "      <td>None</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#23#windSpeed</th>\n",
+       "      <td>None</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#24#pressure</th>\n",
+       "      <td>25800.0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#24#verticalSoundingSignificance</th>\n",
+       "      <td>4</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#24#nonCoordinateGeopotential</th>\n",
+       "      <td>90490.0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#24#airTemperature</th>\n",
+       "      <td>218.5</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#24#dewpointTemperature</th>\n",
+       "      <td>196.5</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#24#windDirection</th>\n",
+       "      <td>None</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#24#windSpeed</th>\n",
+       "      <td>None</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#centre</th>\n",
+       "      <td>98</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#generatingApplication</th>\n",
+       "      <td>1</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "                                        0\n",
+       "#23#pressure                      26300.0\n",
+       "#23#verticalSoundingSignificance        4\n",
+       "#23#nonCoordinateGeopotential     89290.0\n",
+       "#23#airTemperature                  218.5\n",
+       "#23#dewpointTemperature             198.5\n",
+       "#23#windDirection                    None\n",
+       "#23#windSpeed                        None\n",
+       "#24#pressure                      25800.0\n",
+       "#24#verticalSoundingSignificance        4\n",
+       "#24#nonCoordinateGeopotential     90490.0\n",
+       "#24#airTemperature                  218.5\n",
+       "#24#dewpointTemperature             196.5\n",
+       "#24#windDirection                    None\n",
+       "#24#windSpeed                        None\n",
+       "#1#centre                              98\n",
+       "#1#generatingApplication                1"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "df = pdbufr.read_bufr(\"temp.bufr\", columns=\"data\", filters={\"count\": 1}, flat=True)\n",
+    "df.T.iloc[-16:]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Next, we extract the second message. This message contains one more block (25 in total):"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>0</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>#24#pressure</th>\n",
+       "      <td>23200.0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#24#verticalSoundingSignificance</th>\n",
+       "      <td>4</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#24#nonCoordinateGeopotential</th>\n",
+       "      <td>98410.0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#24#airTemperature</th>\n",
+       "      <td>223.1</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#24#dewpointTemperature</th>\n",
+       "      <td>192.1</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#24#windDirection</th>\n",
+       "      <td>None</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#24#windSpeed</th>\n",
+       "      <td>None</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#25#pressure</th>\n",
+       "      <td>20500.0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#25#verticalSoundingSignificance</th>\n",
+       "      <td>4</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#25#nonCoordinateGeopotential</th>\n",
+       "      <td>106300.0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#25#airTemperature</th>\n",
+       "      <td>221.5</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#25#dewpointTemperature</th>\n",
+       "      <td>191.5</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#25#windDirection</th>\n",
+       "      <td>None</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#25#windSpeed</th>\n",
+       "      <td>None</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#centre</th>\n",
+       "      <td>98</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#generatingApplication</th>\n",
+       "      <td>1</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "                                         0\n",
+       "#24#pressure                       23200.0\n",
+       "#24#verticalSoundingSignificance         4\n",
+       "#24#nonCoordinateGeopotential      98410.0\n",
+       "#24#airTemperature                   223.1\n",
+       "#24#dewpointTemperature              192.1\n",
+       "#24#windDirection                     None\n",
+       "#24#windSpeed                         None\n",
+       "#25#pressure                       20500.0\n",
+       "#25#verticalSoundingSignificance         4\n",
+       "#25#nonCoordinateGeopotential     106300.0\n",
+       "#25#airTemperature                   221.5\n",
+       "#25#dewpointTemperature              191.5\n",
+       "#25#windDirection                     None\n",
+       "#25#windSpeed                         None\n",
+       "#1#centre                               98\n",
+       "#1#generatingApplication                 1"
+      ]
+     },
+     "execution_count": 8,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "df = pdbufr.read_bufr(\"temp.bufr\", columns=\"data\", filters={\"count\": 2}, flat=True)\n",
+    "df.T.iloc[-16:]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now, if we extract these messages together the columns will not be aligned:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Warning: not all BUFR messages/subsets have the same structure in the input file. Non-overlapping columns (starting with column[189] = #1#generatingApplication) were added to end of the resulting dataframe altering the original column order for these messages.\n"
+     ]
+    },
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>0</th>\n",
+       "      <th>1</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>#24#pressure</th>\n",
+       "      <td>25800.0</td>\n",
+       "      <td>23200.0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#24#verticalSoundingSignificance</th>\n",
+       "      <td>4</td>\n",
+       "      <td>4</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#24#nonCoordinateGeopotential</th>\n",
+       "      <td>90490.0</td>\n",
+       "      <td>98410.0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#24#airTemperature</th>\n",
+       "      <td>218.5</td>\n",
+       "      <td>223.1</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#24#dewpointTemperature</th>\n",
+       "      <td>196.5</td>\n",
+       "      <td>192.1</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#24#windDirection</th>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#24#windSpeed</th>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#centre</th>\n",
+       "      <td>98</td>\n",
+       "      <td>98</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#generatingApplication</th>\n",
+       "      <td>1</td>\n",
+       "      <td>1</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#25#pressure</th>\n",
+       "      <td>NaN</td>\n",
+       "      <td>20500.0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#25#verticalSoundingSignificance</th>\n",
+       "      <td>NaN</td>\n",
+       "      <td>4.0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#25#nonCoordinateGeopotential</th>\n",
+       "      <td>NaN</td>\n",
+       "      <td>106300.0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#25#airTemperature</th>\n",
+       "      <td>NaN</td>\n",
+       "      <td>221.5</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#25#dewpointTemperature</th>\n",
+       "      <td>NaN</td>\n",
+       "      <td>191.5</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#25#windDirection</th>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#25#windSpeed</th>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "                                        0         1\n",
+       "#24#pressure                      25800.0   23200.0\n",
+       "#24#verticalSoundingSignificance        4         4\n",
+       "#24#nonCoordinateGeopotential     90490.0   98410.0\n",
+       "#24#airTemperature                  218.5     223.1\n",
+       "#24#dewpointTemperature             196.5     192.1\n",
+       "#24#windDirection                    None      None\n",
+       "#24#windSpeed                        None      None\n",
+       "#1#centre                              98        98\n",
+       "#1#generatingApplication                1         1\n",
+       "#25#pressure                          NaN   20500.0\n",
+       "#25#verticalSoundingSignificance      NaN       4.0\n",
+       "#25#nonCoordinateGeopotential         NaN  106300.0\n",
+       "#25#airTemperature                    NaN     221.5\n",
+       "#25#dewpointTemperature               NaN     191.5\n",
+       "#25#windDirection                     NaN       NaN\n",
+       "#25#windSpeed                         NaN       NaN"
+      ]
+     },
+     "execution_count": 9,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "df = pdbufr.read_bufr(\"temp.bufr\", columns=\"data\", filters={\"count\": [1,2]}, flat=True)\n",
+    "df.T.iloc[-16:]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    " So what happened here? The resulting DataFrame was built message by message and columns not yet present were automatically appended to the end by Pandas. We can see that this happened to block #25 from the second message. It changed the original column order because \"#1#centre\" and \"#1#generatingApplication\" now come before and not after block #25. While this is probably a harmless change in this case we can imagine it can pose a significant challenge for more complex message types. \n",
+    " \n",
+    " As a safety measure, when messages are not fully aligned [read_bufr()](../read_bufr.rst) prints a warning message to the stderr.\n",
+    "\n",
+    " To disable the warning message use the **warnings** module as shown below:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>0</th>\n",
+       "      <th>1</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>#24#pressure</th>\n",
+       "      <td>25800.0</td>\n",
+       "      <td>23200.0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#24#verticalSoundingSignificance</th>\n",
+       "      <td>4</td>\n",
+       "      <td>4</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#24#nonCoordinateGeopotential</th>\n",
+       "      <td>90490.0</td>\n",
+       "      <td>98410.0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#24#airTemperature</th>\n",
+       "      <td>218.5</td>\n",
+       "      <td>223.1</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#24#dewpointTemperature</th>\n",
+       "      <td>196.5</td>\n",
+       "      <td>192.1</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#24#windDirection</th>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#24#windSpeed</th>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#centre</th>\n",
+       "      <td>98</td>\n",
+       "      <td>98</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#1#generatingApplication</th>\n",
+       "      <td>1</td>\n",
+       "      <td>1</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#25#pressure</th>\n",
+       "      <td>NaN</td>\n",
+       "      <td>20500.0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#25#verticalSoundingSignificance</th>\n",
+       "      <td>NaN</td>\n",
+       "      <td>4.0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#25#nonCoordinateGeopotential</th>\n",
+       "      <td>NaN</td>\n",
+       "      <td>106300.0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#25#airTemperature</th>\n",
+       "      <td>NaN</td>\n",
+       "      <td>221.5</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#25#dewpointTemperature</th>\n",
+       "      <td>NaN</td>\n",
+       "      <td>191.5</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#25#windDirection</th>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>#25#windSpeed</th>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "                                        0         1\n",
+       "#24#pressure                      25800.0   23200.0\n",
+       "#24#verticalSoundingSignificance        4         4\n",
+       "#24#nonCoordinateGeopotential     90490.0   98410.0\n",
+       "#24#airTemperature                  218.5     223.1\n",
+       "#24#dewpointTemperature             196.5     192.1\n",
+       "#24#windDirection                    None      None\n",
+       "#24#windSpeed                        None      None\n",
+       "#1#centre                              98        98\n",
+       "#1#generatingApplication                1         1\n",
+       "#25#pressure                          NaN   20500.0\n",
+       "#25#verticalSoundingSignificance      NaN       4.0\n",
+       "#25#nonCoordinateGeopotential         NaN  106300.0\n",
+       "#25#airTemperature                    NaN     221.5\n",
+       "#25#dewpointTemperature               NaN     191.5\n",
+       "#25#windDirection                     NaN       NaN\n",
+       "#25#windSpeed                         NaN       NaN"
+      ]
+     },
+     "execution_count": 10,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "import warnings\n",
+    "warnings.filterwarnings(\"ignore\", module=\"pdbufr\")\n",
+    "\n",
+    "df = pdbufr.read_bufr(\"temp.bufr\", columns=\"data\", filters={\"count\": [1,2]}, flat=True)\n",
+    "df.T.iloc[-16:]"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3.8.14 ('mpy38')",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.15"
+  },
+  "orig_nbformat": 4,
+  "vscode": {
+   "interpreter": {
+    "hash": "22dc05efe0944894879e71a134ce5db002aedecbcd8b98acee6e3c2217e44519"
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/docs/index.rst b/docs/index.rst
index 7d06de5..55f4eab 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -6,8 +6,7 @@ Welcome to pdbufr's documentation
 
   This documentation is still work in progress and can only be regarded as a **draft**.
 
-*pdbufr* is a Python package implementing a `Pandas <https://pandas.pydata.org>`_ reader for the BUFR format using  `ecCodes <https://confluence.ecmwf.int/display/ECC>`_.  It features the :func:`read_bufr` function to extract data from a BUFR file as a Pandas DataFrame
-using a rich filtering engine.
+*pdbufr* is a Python package implementing a `Pandas <https://pandas.pydata.org>`_ reader for the BUFR format using  `ecCodes <https://confluence.ecmwf.int/display/ECC>`_.  It features the :func:`read_bufr` function to extract data from a BUFR file as a Pandas DataFrame using a rich filtering engine.
 
 *pdbufr* supports BUFR 3 and 4 files with uncompressed and compressed subsets. It works on Linux, MacOS and Windows, the ecCodes C-library is the only binary dependency. All modern versions of Python (>=3.6) and PyPy3 are supported.
 
diff --git a/docs/install.rst b/docs/install.rst
index ab73888..92c1c2d 100644
--- a/docs/install.rst
+++ b/docs/install.rst
@@ -1,33 +1,43 @@
 Installation
 ============
 
-The easiest way to install *pdbufr* dependencies is via Conda::
+Installing pdbufr
+--------------------
 
-    $ conda install -c conda-forge python-eccodes pandas
+*pdbufr* can be installed from PyPI with:
 
-and *pdbufr* itself as a Python package from PyPI with::
+.. code-block:: bash
 
-    $ pip install pdbufr
+    pip install pdbufr
 
-System dependencies
--------------------
+
+Installing the Python module
+-------------------------------
 
 The Python module depends on the ECMWF *ecCodes* library
-that must be installed on the system and accessible as a shared library.
-Some Linux distributions ship a binary version that may be installed with the standard package manager.
-On Ubuntu 18.04 use the command::
+that must be installed on the system and accessible as a shared library. The easiest way to install it is to use Conda:
+
+.. code-block:: bash
+
+    conda install eccodes -c conda-forge
 
-    $ sudo apt-get install libeccodes0
 
-On a MacOS with HomeBrew use::
+On a MacOS it is also available from HomeBrew:
 
-    $ brew install eccodes
+.. code-block:: bash
+
+    brew install eccodes
 
 As an alternative you may install the official source distribution
 by following the instructions at
 https://software.ecmwf.int/wiki/display/ECC/ecCodes+installation
 
-You may run a simple selfcheck command to ensure that your system is set up correctly::
+Selfcheck
+------------
+
+You may run a simple selfcheck command to ensure that your system is set up correctly:
+
+.. code-block:: bash
 
     $ python -m pdbufr selfcheck
     Found: ecCodes v2.19.0.
diff --git a/docs/read_bufr.rst b/docs/read_bufr.rst
index aa9528a..4d148c4 100644
--- a/docs/read_bufr.rst
+++ b/docs/read_bufr.rst
@@ -1,31 +1,48 @@
 read_bufr
-==========
+==============
 
-..  py:function:: read_bufr(path, columns, filters={}, required_columns=True)
+.. py:function:: read_bufr(path, columns=[], filters={}, required_columns=True, flat=False)
+
+    Extract data from BUFR as a pandas.DataFrame with the specified ``columns`` applying the ``filters`` either in :ref:`hierarchical <tree-mode-section>` or :ref:`flat <flat-mode-section>` mode.
+    
+    :param path: path to the BUFR file
+    :type path: str, bytes, os.PathLike
+    :param columns: a list of ecCodes BUFR keys to extract for each BUFR message/subset. When ``flat`` is True ``columns`` must be one of the following string values: 
+      
+          * "all", empty str or empty list (default): all the columns are extracted
+          * "header": only the columns from the header section are extracted
+          * "data": only the columns from the data section are extracted
+
+    :type columns: str, sequence[str]
+    :param filters: defines the conditions when to extract the specified ``columns``. The individual conditions are combined together with the logical AND operator to form the filter. See :ref:`filters-section` for details.
+    :type filters: dict
+    :param required_columns: the list of ecCodes BUFR keys that are required to be present in the BUFR message/subset. Bool values are interpreted as follows:
+
+        * if ``flat`` is False:
+    
+          * True means all the keys in ``columns`` are required
+          * False means no columns are required
+
+        * if ``flat`` is True either bool value means no columns are required
+    
+    :type required_columns: bool, iterable[str]
+    :param flat: enables flat extraction mode. When it is ``True`` each message/subset is treated as a :ref:`flat list <flat-mode-section>`, while when it is ``False`` (default), data is extracted as if the message had a :ref:`tree-like hierarchy <tree-mode-section>`. See details below. New in *version 0.10.0*
+    :type flat: bool
+    :rtype: pandas.DataFrame
 
-   Extract data from BUFR as a pandas.DataFrame with the specified ``columns`` applying the ``filters``.
-   
-   :param path: path to the BUFR file
-   :type path: str, bytes, os.PathLike
-   :param columns: A list of ecCodes BUFR keys to extract for each BUFR message/subset.
-   :type columns: iterable
-   :param filters: A dictionary of ecCodes BUFR key filter conditions. The individual conditions are combined together with the logical AND operator to form the filter. See details below.
-   :type filters: dict
-   :param required_columns: The list of ecCodes BUFR keys that are required to be present in the BUFR message/subset. ``True`` means all the keys in ``columns`` are required.
-   :type required_columns: bool, iterable[str]
-   :rtype: pandas.DataFrame
 
+    In order to correctly use :func:`read_bufr` for a given BUFR file first you need to understand the structure of the messages and the keys/values you can use for data extraction and filter definition. The BUFR structure can be explored with *ecCodes* command line tools `bufr_ls <https://confluence.ecmwf.int/display/ECC/bufr_ls>`_  and `bufr_dump <https://confluence.ecmwf.int/display/ECC/bufr_dump>`_. You can also use `CodesUI <https://confluence.ecmwf.int/display/METV/CodesUI>`_ or `Metview <https://metview.readthedocs.io>`_, which provide graphical user interfaces to inspect BUFR/GRIB data.
 
-   In order to correctly use :func:`read_bufr` for a given BUFR file first you need to understand the structure of the messages and the keys/values you can use for data extraction and filter definition. The BUFR structure can be explored with *ecCodes* command line tools `bufr_ls <https://confluence.ecmwf.int/display/ECC/bufr_ls>`_  and  `bufr_dump <https://confluence.ecmwf.int/display/ECC/bufr_dump>`_.
+    There are some :ref:`notebook examples <examples>` available demonstrating how to use :func:`read_bufr` for various observation/forecast BUFR data types. 
 
-   There are some :ref:`notebook examples <examples>` available demonstrating how to use :func:`read_bufr` for various observation/forecast BUFR data types. 
 
-   **Keys**
+BUFR keys 
+-----------
 
-   ecCodes keys from both the BUFR header and data sections are supported but there are some limitations:
+   ecCodes keys from both the BUFR header and data sections are supported in ``columns``, ``filters`` and ``required_columns``. However, there are some limitations:
    
-     * keys containing the rank e.g. "#1#latitude#" cannot be used
-     * key attributes e.g. "latitude->code" cannot be used
+        * keys containing the rank e.g. "#1#latitude" cannot be used
+        * key attributes e.g. "latitude->code" cannot be used
   
    The "count" generated key, which refers to the message index, is also supported but please note that message indexing starts at 1 and not at 0!
    
@@ -73,11 +90,16 @@ read_bufr
 
           The computed keys do not preserve their position in ``columns`` but are placed to the end of the resulting DataFrame.
 
-    **Filters** 
+.. _filters-section:
+
+Filters
+--------------
+
+    The filter conditions are specified as a dict via ``filters`` and determine when the specified ``columns`` will actually be extracted.
 
     A filter condition can be a single value match:
 
-     .. code-block:: python 
+      .. code-block:: python 
 
           filters={"blockNumber": 12}
 
@@ -90,7 +112,7 @@ read_bufr
           
     or an interval expressed as a ``slice`` (the boundaries as inclusive):
 
-     .. code-block:: python
+      .. code-block:: python
                
           # closed interval (>=273.16 and <=293.16)  
           filters={"airTemperature": slice(273.16, 293.16)}
@@ -125,27 +147,106 @@ read_bufr
                slice(datetime.datetime(2009,1,23,13,0), 
                      datetime.datetime(2009,1,23,13,1))}
 
+.. _tree-mode-section:
+
+Hierarchical mode
+-------------------
     
-    **Algorithm**
-
-    A BUFR message/subset seemingly has a flat structure but actually it can be interpreted as a hierarchy. According to the `WMO BUFR manual <https://community.wmo.int/activity-areas/wmo-codes/manual-codes/bufr-edition-3-and-crex-edition-1>`_ each key in class 01-09 introduces a new hierarchy level in the BUFR message/subset::
-
-          Element descriptors corresponding to the following classes in Table B 
-          shall remain in effect until superseded by redefinition:
-               Class
-               01 Identification
-               02 Instrumentation
-               03 Instrumentation
-               04 Location (time)
-               05 Location (horizontal - 1)
-               06 Location (horizontal - 2)
-               07 Location (vertical)
-               08 Significance qualifiers
-               09 Reserved
-               
-          Note: Redefinition is effected by the occurrence of element descriptors
-               which contradict the preceding element descriptors from these classes. If two or
-               more elements from the same class do not contradict one another, they all apply.
+    When ``flat`` is ``False`` the contents of a BUFR message/subset is interpreted as a hierarchical structure. This is based on a certain group of BUFR keys (related to instrumentation, location etc), which according to the `WMO BUFR manual <https://community.wmo.int/activity-areas/wmo-codes/manual-codes/bufr-edition-3-and-crex-edition-1>`_ introduce a new hierarchy level in the message/susbset. During data extraction ``read_bufr`` traverses this hierarchy and when all the ``columns`` are collected and the all the ``filters`` match a new record is added to the output. With this several records can be extracted from the same message/subset.
 
-     
-    This may sound cryptic but this is what ``read_bufr`` uses to define the hierarchy and decide when a set of collected columns has to be added to the output as a new record.
+    **Example**
+      
+    In this example we extract values from a classic radiosonde observation BUFR file. Here each message contains a single location ("latitude", "longitude") with several pressure levels of temperature, dewpoint etc. The message hierarchy is shown in the following snapshot:
+
+      .. image:: /_static/temp_structure.png
+          :width: 450px  
+
+    To extract the temperature profile for the first two stations we can use this code:
+
+      .. code-block:: python
+
+          df = pdbufr.read_bufr("temp.bufr", 
+          columns=("latitude", "longitude", "pressure", "airTemperature"),
+          filters={"count": [1, 2]}, 
+          )
+
+    which results in the following DataFrame:
+
+      .. literalinclude:: _static/h_dump_output.txt
+
+
+.. _flat-mode-section:
+
+Flat mode
+--------------
+
+    New in *version 0.10.0*
+
+    When ``flat`` is ``True`` messages/subsets are extracted as a whole preserving the column order (see the note below for exceptions) and each extracted message/subset will be a separate record in the resulting DataFrame.
+    
+    With ``filters`` we can control which messages/subsets should be selected. By default, all the columns in a message/subset are extracted (see the exceptions below), but this can be changed by setting ``columns`` to "header" or "data" to get only the header or data section keys. Other column selection modes are not available.
+    
+    In the resulting DataFrame the original ecCodes keys containing the **rank** are used as column names, e.g. "#1#latitude" instead of "latitude". The following set of keys are omitted:
+
+    * from the header: "unexpandedDescriptors"
+    * from the data section: data description operator qualifiers  (e.g. "delayedDescriptorReplicationFactor") and "operator"
+    * key attributes e.g. "latitude->code"
+
+    The **rank** appearing in the keys in a message containing **uncompressed subsets** is not reset by ecCodes when a new subset started. To make the columns as aligned as a possible in the output :func:`read_bufr` resets the rank and ensures that e.g. the first "latitude" key is always called "#1#latitude" in each uncompressed subset.
+
+    ``filters`` control what messages/subsets should be extracted from the BUFR file. They are interpreted in a different way than in the  :ref:`hierarchical <tree-mode-section>` mode:
+
+    * they can only contain keys without a rank
+    * for **non-computed keys** the filter condition matches if there is a match for the same key with any given rank in the message/subset. E.g. if ::
+
+        filters = {"pressure": 50000}
+
+      and there is e.g. a value "#12#pressure" = 50000 in the message/subset then the filter matches.
+    * for **computed keys** the filter condition matches if there is a match for the involved keys at their first occurrence (e.i. rank=1) in the message/subset. E.g::
+ 
+         filters = {"WMO_station_id": 12925}
+
+      matches if "#1#blockNumber" = 12 and "#1#stationNumber" = 925 in the message/subset (remember WMO_station_id=blockNumber*1000+stationNumber)
+
+    .. warning::
+
+        Messages/subsets in a BUFR file can have a different set of BUFR keys. When a new message/subset is processed :func:`read_bufr` adds it to the resulting DataFrame as a new record and columns that are not yet present in the output are automatically appended by Pandas to the end changing the original order of keys for that message. When this happens :func:`pdbufr` prints a warning message to the stdout
+        (see the example below or the :ref:`/examples/flat_dump.ipynb` notebook for details).
+        
+    **Example**
+
+    We use the same radiosonde BUFR file as for the :ref:`hierarchical mode <tree-mode-section>` example above. To extract all the data values for the first two stations we can use this code:
+
+      .. code-block:: python
+  
+        df = pdbufr.read_bufr("temp.bufr", columns="data",
+                flat=True
+                filters={"count": [1, 2]},  
+              )
+
+    which results in the following DataFrame:
+
+      .. literalinclude:: _static/flat_dump_output.txt
+
+    and generates the following warning::
+
+      Warning: not all BUFR messages/subsets have the same structure in the input file.
+      Non-overlapping columns (starting with column[189] = #1#generatingApplication) 
+      were added to end of the resulting dataframe altering the original column order
+      for these messages.
+
+    This warning can be disabled by using the **warnings** module. The code below produces the same DataFrame as the one above but does not print the warning message:
+
+      .. code-block:: python
+  
+        import warnings
+        warnings.filterwarnings("ignore", module="pdbufr")
+
+        df = pdbufr.read_bufr("temp.bufr", columns="data",
+              flat=True
+              filters={"count": [1, 2]},  
+            )
+
+    .. note::
+
+      See the :ref:`/examples/flat_dump.ipynb` notebook for more details.
diff --git a/pdbufr/bufr_read.py b/pdbufr/bufr_read.py
index cd8103c..3462dc0 100644
--- a/pdbufr/bufr_read.py
+++ b/pdbufr/bufr_read.py
@@ -17,21 +17,56 @@
 
 def read_bufr(
     path: T.Union[str, bytes, "os.PathLike[T.Any]"],
-    columns: T.Iterable[str],
+    columns: T.Union[T.Sequence[str], str] = [],
     filters: T.Mapping[str, T.Any] = {},
     required_columns: T.Union[bool, T.Iterable[str]] = True,
+    flat: bool = False,
 ) -> pd.DataFrame:
     """
     Read selected observations from a BUFR file into DataFrame.
-
-    :param path: The path to the BUFR file
-    :param columns: A list of BUFR keys to return in the DataFrame for every observation
-    :param filters: A dictionary of BUFR key / filter definition to filter the observations to return
-    :param required_columns: The list BUFR keys that are required for all observations.
-        ``True`` means all ``columns`` are required
     """
+
     with BufrFile(path) as bufr_file:  # type: ignore
-        observations = bufr_structure.stream_bufr(
-            bufr_file, columns, filters, required_columns
-        )
-        return pd.DataFrame.from_records(observations)
+        if not flat:
+            observations = bufr_structure.stream_bufr(
+                bufr_file, columns, filters=filters, required_columns=required_columns
+            )
+            return pd.DataFrame.from_records(observations)
+        else:
+
+            class ColumnInfo:
+                def __init__(self) -> None:
+                    self.first_count = 0
+
+            column_info = ColumnInfo()
+
+            # returns a generator
+            observations = bufr_structure.stream_bufr_flat(
+                bufr_file,
+                columns,
+                filters=filters,
+                required_columns=required_columns,
+                column_info=column_info,
+            )
+
+            df = pd.DataFrame.from_records(observations)
+
+            # compare the column count in the first record to that of the
+            # dataframe. If the latter is larger, then there were non-aligned columns,
+            # which were appended to the end of the dataframe columns.
+            if column_info.first_count > 0 and column_info.first_count < len(
+                df.columns
+            ):
+                import warnings
+
+                # temporarily overwrite warnings formatter
+                ori_formatwarning = warnings.formatwarning
+                warnings.formatwarning = (
+                    lambda msg, *args, **kwargs: f"Warning: {msg}\n"
+                )
+                warnings.warn(
+                    f"not all BUFR messages/subsets have the same structure in the input file. Non-overlapping columns (starting with column[{column_info.first_count-1}] = {df.columns[column_info.first_count-1]}) were added to end of the resulting dataframe altering the original column order for these messages."
+                )
+                warnings.formatwarning = ori_formatwarning
+
+            return df
diff --git a/pdbufr/bufr_structure.py b/pdbufr/bufr_structure.py
index 1c47420..88463ca 100644
--- a/pdbufr/bufr_structure.py
+++ b/pdbufr/bufr_structure.py
@@ -8,6 +8,7 @@
 
 import collections
 import datetime
+import re
 import typing as T
 
 import attr
@@ -19,6 +20,14 @@
 from . import bufr_filters
 
 
+def rank_from_key(key: str) -> int:
+    rank_text, sep, _ = key.rpartition("#")
+    if sep == "#":
+        return int(rank_text[1:])
+    else:
+        return 0
+
+
 @attr.attrs(auto_attribs=True, frozen=True)
 class BufrKey:
     level: int
@@ -43,6 +52,41 @@ def key(self) -> str:
         return prefix + self.name
 
 
+@attr.attrs(auto_attribs=True)
+class UncompressedBufrKey:
+    current_rank: int
+    ref_rank: int
+    name: str
+
+    @classmethod
+    def from_key(cls, key: str) -> "UncompressedBufrKey":
+        rank_text, sep, name = key.rpartition("#")
+        try:
+            if sep == "#":
+                rank = int(rank_text[1:])
+            else:
+                rank = 0
+        except:
+            rank = 0
+
+        return cls(rank, 0, name)
+
+    def update_rank(self, key: str) -> None:
+        self.current_rank = rank_from_key(key)
+
+    def adjust_ref_rank(self) -> None:
+        self.ref_rank = self.current_rank
+
+    @property
+    def relative_key(self) -> str:
+        if self.current_rank > 0:
+            rel_rank = self.current_rank - self.ref_rank
+            prefix = f"#{rel_rank}#"
+        else:
+            prefix = ""
+        return prefix + self.name
+
+
 IS_KEY_COORD = {"subsetNumber": True, "operator": False}
 
 
@@ -254,6 +298,7 @@ def extract_observations(
         current_observation = collections.OrderedDict(base_observation)
         current_levels: T.List[int] = [0]
         failed_match_level: T.Optional[int] = None
+
         for bufr_key in filtered_keys:
             level = bufr_key.level
             name = bufr_key.name
@@ -310,6 +355,146 @@ def extract_observations(
             yield dict(current_observation)
 
 
+def extract_message(
+    message: T.Mapping[str, T.Any],
+    filters: T.Dict[str, bufr_filters.BufrFilter] = {},
+    base_observation: T.Dict[str, T.Any] = {},
+    required_columns: T.Set[str] = set(),
+    header_keys: T.Set[str] = set(),
+) -> T.Iterator[T.Dict[str, T.Any]]:
+    value_cache = {}
+    try:
+        is_compressed = bool(message["compressedData"])
+    except (KeyError, eccodes.KeyValueNotFoundError):
+        is_compressed = False
+
+    if is_compressed:
+        is_uncompressed = False
+        subset_count = message["numberOfSubsets"]
+    else:
+        is_uncompressed = int(message["numberOfSubsets"]) > 1
+        subset_count = 1
+
+    # For messages with uncompressed subsets consider this:
+    # - for each data key we have a single value
+    # - there is no way to identify the subset from the key
+    # - we cannot directly iterate over a given subset
+    # - if we iterate over the keys a new subset is indicated by the
+    #   appearance of the "subsetNumber" key, which contains the same array
+    #   of values each time (the subset index for all the subsets). This key is
+    #   generated by ecCodes and does not contain any ranking so its name is
+    #   always "subsetNumber".
+
+    uncompressed_keys: T.Dict["str", T.Any]
+    uncompressed_keys = dict()
+    skip_keys = {
+        "unexpandedDescriptors",
+        "shortDelayedDescriptorReplicationFactor",
+        "delayedDescriptorReplicationFactor",
+        "extendedDelayedDescriptorReplicationFactor",
+        "delayedDescriptorAndDataRepetitionFactor",
+        "extendedDelayedDescriptorAndDataRepetitionFactor"
+        "associatedFieldSignificance",
+        "dataPresentIndicator",
+        "operator",
+    }
+
+    for subset in range(subset_count):
+        filters_match = {k: False for k in filters.keys()}
+        required_columns_match = {k: False for k in required_columns}
+        current_observation: T.Dict[str, T.Any]
+        current_observation = collections.OrderedDict(base_observation)
+
+        uncompressed_subset = 0
+        for key in message:
+
+            name = key.rpartition("#")[2]
+            if name in skip_keys or "->" in key:
+                continue
+
+            if is_uncompressed and key == "subsetNumber":
+                if uncompressed_subset > 0:
+                    if (
+                        current_observation
+                        and all(filters_match.values())
+                        and all(required_columns_match.values())
+                    ):
+                        yield dict(current_observation)
+
+                    # header keys appear only once so we need to keep the match info for them
+                    for k in filters_match:
+                        if k not in header_keys:
+                            filters_match[k] = False
+
+                    for k in required_columns_match:
+                        if k not in header_keys:
+                            required_columns_match[k] = False
+
+                    # only keep the header keys in the result because they are the
+                    # same for all the subsets
+                    while (
+                        len(current_observation)
+                        and "subsetNumber" in current_observation
+                    ):
+                        # OrderedDict.popitem uses LIFO orde
+                        current_observation.popitem()
+
+                    for v in uncompressed_keys.values():
+                        v.adjust_ref_rank()
+
+                uncompressed_subset += 1
+
+            if is_compressed:
+                if key not in value_cache:
+                    value_cache[key] = message[key]
+                value = value_cache[key]
+            else:
+                value = message[key]
+
+            # extract compressed BUFR values. They are either numpy arrays (for numeric types)
+            # or lists of strings
+            if (
+                is_compressed
+                and name != "unexpandedDescriptors"
+                and isinstance(value, (np.ndarray, list))
+                and len(value) == subset_count
+            ):
+                value = value[subset]
+
+            if isinstance(value, float) and value == eccodes.CODES_MISSING_DOUBLE:
+                value = None
+            elif isinstance(value, int) and value == eccodes.CODES_MISSING_LONG:
+                value = None
+
+            if is_uncompressed:
+                if name not in uncompressed_keys:
+                    uncompressed_keys[name] = UncompressedBufrKey.from_key(key)
+                else:
+                    uncompressed_keys[name].update_rank(key)
+                key = uncompressed_keys[name].relative_key
+
+                # subsetNumber is an array and we need the current value
+                if key == "subsetNumber":
+                    value = uncompressed_subset
+
+            if name in filters:
+                if filters[name].match(value):
+                    filters_match[name] = True
+
+            if name in required_columns:
+                required_columns_match[name] = True
+
+            current_observation[key] = value
+
+        # yield the last observation
+        if (
+            current_observation
+            and all(filters_match.values())
+            and all(required_columns_match.values())
+        ):
+            yield dict(current_observation)
+
+
 def add_computed_keys(
     observation: T.Dict[str, T.Any],
     included_keys: T.Container[str],
@@ -333,6 +518,26 @@ def add_computed_keys(
     return augmented_observation
 
 
+def test_computed_keys(
+    observation: T.Dict[str, T.Any],
+    filters: T.Dict[str, bufr_filters.BufrFilter] = {},
+    prefix: str = "",
+) -> bool:
+    for keys, computed_key, getter in COMPUTED_KEYS:
+        if computed_key in filters:
+            computed_value = None
+            try:
+                computed_value = getter(observation, prefix, keys)
+            except Exception:
+                return False
+            if computed_value is not None:
+                if not filters[computed_key].match(computed_value):
+                    return False
+            else:
+                return False
+    return True
+
+
 class CMWrapper:
     """Makes it possible to use context manager both with BufrMessage and dict type of messages"""
 
@@ -352,7 +557,7 @@ def __exit__(self, *args) -> None:  # type: ignore
 
 def stream_bufr(
     bufr_file: T.Iterable[T.MutableMapping[str, T.Any]],
-    columns: T.Iterable[str],
+    columns: T.Union[T.Sequence[str], str],
     filters: T.Mapping[str, T.Any] = {},
     required_columns: T.Union[bool, T.Iterable[str]] = True,
     prefilter_headers: bool = False,
@@ -438,3 +643,127 @@ def stream_bufr(
             # optimisation: skip decoding messages above max_count
             if max_count is not None and count >= max_count:
                 break
+
+
+def stream_bufr_flat(
+    bufr_file: T.Iterable[T.MutableMapping[str, T.Any]],
+    columns: T.Union[T.Sequence[str], str],
+    filters: T.Mapping[str, T.Any] = {},
+    required_columns: T.Union[bool, T.Iterable[str]] = True,
+    prefilter_headers: bool = False,
+    column_info: T.Any = None,
+) -> T.Iterator[T.Dict[str, T.Any]]:
+
+    if isinstance(columns, str):
+        columns = (columns,)
+
+    if not isinstance(columns, T.Sequence):
+        raise TypeError("invalid columns type")
+    elif len(columns) == 0 or columns[0] == "":
+        columns = ("all",)
+    elif len(columns) != 1:
+        raise ValueError(f"when columns is an iterable it can have maximum 1 element")
+
+    if columns[0] not in ["all", "header", "data"]:
+        raise ValueError(f"columns must be all, header or data")
+
+    add_header = columns[0] in ["all", "header"]
+    add_data = columns[0] in ["all", "data"]
+    if not add_header and not add_data:
+        raise ValueError("either header or data must be extracted")
+
+    if isinstance(required_columns, bool):
+        required_columns = set()
+    elif isinstance(required_columns, str):
+        required_columns = {required_columns}
+    elif isinstance(required_columns, T.Iterable):
+        required_columns = set(required_columns)
+    else:
+        raise TypeError("required_columns must be a bool, str or an iterable")
+
+    # compile filters
+    filters = dict(filters)
+    value_filters = {k: bufr_filters.BufrFilter.from_user(filters[k]) for k in filters}
+
+    # prepare count filter
+    if "count" in value_filters:
+        max_count = value_filters["count"].max()
+    else:
+        max_count = None
+
+    count_filter = value_filters.pop("count", None)
+
+    # prepare computed keys
+    computed_keys = [x for _, x, _ in COMPUTED_KEYS]
+    value_filters_without_computed = {
+        k: v for k, v in value_filters.items() if k not in computed_keys
+    }
+
+    if column_info is not None:
+        column_info.first_count = 0
+
+    for count, msg in enumerate(bufr_file, 1):
+        # we use a context manager to automatically delete the handle of the BufrMessage.
+        # We have to use a wrapper object here because a message can also be a dict
+        with CMWrapper(msg) as message:
+            # count filter
+            if count_filter is not None and not count_filter.match(count):
+                continue
+
+            message_value_filters = value_filters_without_computed
+            message_required_columns = required_columns
+
+            header_keys = set()
+            if not add_header or prefilter_headers or value_filters or required_columns:
+                header_keys = set(message.keys())
+
+                if required_columns:
+                    message_required_columns = required_columns - set(header_keys)
+
+                # test header keys for failed matches before unpacking
+                if prefilter_headers:
+                    if not bufr_filters.is_match(
+                        message, value_filters, required=False
+                    ):
+                        continue
+                    # remove already tested filters
+                    else:
+                        message_value_filters = {
+                            k: v
+                            for k, v in value_filters.items()
+                            if k not in header_keys
+                        }
+
+            message["skipExtraKeyAttributes"] = 1
+
+            if add_data or message_value_filters or message_required_columns:
+                message["unpack"] = 1
+
+            observation: T.Dict[str, T.Any] = {}
+
+            for observation in extract_message(
+                message,
+                message_value_filters,
+                observation,
+                message_required_columns,
+                header_keys,
+            ):
+                if header_keys:
+                    if not add_header:
+                        for key in header_keys:
+                            observation.pop(key, None)
+                    if not add_data:
+                        data_keys = set(observation.keys()) - header_keys
+                        for key in data_keys:
+                            observation.pop(key, None)
+
+                if observation and test_computed_keys(
+                    observation, value_filters, "#1#"
+                ):
+                    if column_info is not None and column_info.first_count == 0:
+                        column_info.first_count = len(observation)
+                    yield observation
+
+            # optimisation: skip decoding messages above max_count
+            if max_count is not None and count >= max_count:
+                break


Test patch:
diff --git a/tests/ref_data/obs_3day_ref_1.csv b/tests/ref_data/obs_3day_ref_1.csv
new file mode 100644
index 0000000..37f2ddb
--- /dev/null
+++ b/tests/ref_data/obs_3day_ref_1.csv
@@ -0,0 +1,2 @@
+edition,masterTableNumber,bufrHeaderSubCentre,bufrHeaderCentre,updateSequenceNumber,dataCategory,dataSubCategory,masterTablesVersionNumber,localTablesVersionNumber,typicalYearOfCentury,typicalMonth,typicalDay,typicalHour,typicalMinute,typicalDate,typicalTime,rdbType,oldSubtype,localYear,localMonth,localDay,localHour,localMinute,localSecond,rdbtimeDay,rdbtimeHour,rdbtimeMinute,rdbtimeSecond,rdbtimeTime,rectimeDay,rectimeHour,rectimeMinute,rectimeSecond,restricted,correction1,correction1Part,correction2,correction2Part,correction3,correction3Part,correction4,correction4Part,qualityControl,newSubtype,numberOfSubsets,localLatitude,localLongitude,ident,observedData,compressedData,subsetNumber,#1#blockNumber,#1#stationNumber,#1#stationType,#1#year,#1#month,#1#day,#1#hour,#1#minute,#1#latitude,#1#longitude,#1#heightOfStation,#1#nonCoordinatePressure,#1#pressureReducedToMeanSeaLevel,#1#3HourPressureChange,#1#characteristicOfPressureTendency,#1#windDirectionAt10M,#1#windSpeedAt10M,#1#airTemperatureAt2M,#1#dewpointTemperatureAt2M,#1#relativeHumidity,#1#horizontalVisibility,#1#presentWeather,#1#pastWeather1,#1#pastWeather2,#1#cloudCoverTotal,#1#verticalSignificanceSurfaceObservations,#1#cloudAmount,#1#heightOfBaseOfCloud,#1#cloudType,#2#cloudType,#3#cloudType,#2#verticalSignificanceSurfaceObservations,#2#cloudAmount,#4#cloudType,#2#heightOfBaseOfCloud,#3#verticalSignificanceSurfaceObservations,#3#cloudAmount,#5#cloudType,#3#heightOfBaseOfCloud,#4#verticalSignificanceSurfaceObservations,#4#cloudAmount,#6#cloudType,#4#heightOfBaseOfCloud,#5#verticalSignificanceSurfaceObservations,#5#cloudAmount,#7#cloudType,#5#heightOfBaseOfCloud,#1#totalPrecipitationPast6Hours,#1#totalSnowDepth,#1#centre,#1#generatingApplication
+3,0,0,98,1,0,1,13,1,17,4,25,12,0,20170425,120000,1,1,2017,4,25,12,0,0,25,11,53,46,115346,10,10,0,0,0,0,1,0,0,0,0,0,0,70,0,1,52.12,0.96,03590,1,0,1,3,590,1,2017,4,25,12,0,52.120000000000005,0.9600000000000001,90,100170.0,101270.0,-10.0,8,300,7.0,281.7,268.40000000000003,,50000.0,2,1,1,40,1,3,1250.0,31,20,10,1,3,8,1460.0,2,,,,3,,,,,,,,0.0,,98,1
diff --git a/tests/ref_data/synop_uncompressed_ref_1.csv b/tests/ref_data/synop_uncompressed_ref_1.csv
new file mode 100644
index 0000000..7d0c742
--- /dev/null
+++ b/tests/ref_data/synop_uncompressed_ref_1.csv
@@ -0,0 +1,3 @@
+edition,masterTableNumber,bufrHeaderCentre,bufrHeaderSubCentre,updateSequenceNumber,dataCategory,internationalDataSubCategory,dataSubCategory,masterTablesVersionNumber,localTablesVersionNumber,typicalYear,typicalMonth,typicalDay,typicalHour,typicalMinute,typicalSecond,typicalDate,typicalTime,numberOfSubsets,observedData,compressedData,subsetNumber,#1#blockNumber,#1#stationNumber,#1#stationOrSiteName,#1#stationType,#1#year,#1#month,#1#day,#1#hour,#1#minute,#1#latitude,#1#longitude,#1#heightOfStationGroundAboveMeanSeaLevel,#1#heightOfBarometerAboveMeanSeaLevel,#1#nonCoordinatePressure,#1#pressureReducedToMeanSeaLevel,#1#3HourPressureChange,#1#characteristicOfPressureTendency,#1#24HourPressureChange,#1#pressure,#1#nonCoordinateGeopotentialHeight,#1#heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform,#1#airTemperature,#1#dewpointTemperature,#1#relativeHumidity,#2#heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform,#1#horizontalVisibility,#3#heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform,#1#totalPrecipitationPast24Hours,#4#heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform,#1#cloudCoverTotal,#1#verticalSignificanceSurfaceObservations,#1#cloudAmount,#1#heightOfBaseOfCloud,#1#cloudType,#2#cloudType,#3#cloudType,#2#verticalSignificanceSurfaceObservations,#1#stateOfGround,#1#totalSnowDepth,#1#groundMinimumTemperaturePast12Hours,#1#presentWeather,#1#timePeriod,#1#pastWeather1,#1#pastWeather2,#2#timePeriod,#1#totalSunshine,#3#timePeriod,#2#totalSunshine,#5#heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform,#4#timePeriod,#1#totalPrecipitationOrTotalWaterEquivalent,#5#timePeriod,#2#totalPrecipitationOrTotalWaterEquivalent,#6#heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform,#6#timePeriod,#7#timePeriod,#1#maximumTemperatureAtHeightAndOverPeriodSpecified,#8#timePeriod,#9#timePeriod,#1#minimumTemperatureAtHeightAndOverPeriodSpecified,#7#heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform,#1#instrumentationForWindMeasurement,#1#timeSignificance,#10#timePeriod,#1#windDirection,#1#windSpeed,#2#timeSignificance,#11#timePeriod,#1#maximumWindGustDirection,#1#maximumWindGustSpeed,#12#timePeriod,#2#maximumWindGustDirection,#2#maximumWindGustSpeed,#8#heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform,#13#timePeriod,#1#typeOfInstrumentationForEvaporationMeasurement,#1#evaporation,#14#timePeriod,#1#maximumWindSpeed10MinuteMeanWind
+4,0,88,0,0,0,0,0,14,0,2015,1,26,10,0,0,20150126,100000,12,1,0,1,1,27,TROMSO-HOLT,0,2015,1,26,10,0,69.65230000000001,18.905700000000003,20.0,,,,,,,,,2.0,276.45,271.39,69,,,2.0,,,,,,,,,,,,,,,,,,,,,,2.0,,,-1,0.0,2.0,-1,0,276.55,-1,0,276.25,2.0,8,2,-10,,,,,,,,,,,,,,,
+4,0,88,0,0,0,0,0,14,0,2015,1,26,10,0,0,20150126,100000,12,1,0,2,1,84,PASVIK,0,2015,1,26,10,0,69.4552,30.041000000000004,27.0,,,,,,,,,2.0,266.55,264.83,86,,,2.0,,,,,,,,,,,,,,,,,,,,,,2.0,,,-1,0.0,2.0,-1,0,266.55,-1,0,265.45,10.0,8,2,-10,,,,,,,,,,,,,,,
diff --git a/tests/sample-data/pgps_110.bufr b/tests/sample-data/pgps_110.bufr
new file mode 100644
index 0000000..259b422
Binary files /dev/null and b/tests/sample-data/pgps_110.bufr differ
diff --git a/tests/test_60_flat_mode.py b/tests/test_60_flat_mode.py
new file mode 100644
index 0000000..120d6fc
--- /dev/null
+++ b/tests/test_60_flat_mode.py
@@ -0,0 +1,684 @@
+# (C) Copyright 2019- ECMWF.
+#
+# This software is licensed under the terms of the Apache Licence Version 2.0
+# which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
+# In applying this licence, ECMWF does not waive the privileges and immunities
+# granted to it by virtue of its status as an intergovernmental organisation
+# nor does it submit to any jurisdiction.
+
+import os
+import typing as T
+import warnings
+
+import numpy as np
+import pytest
+
+import pdbufr
+
+pd = pytest.importorskip("pandas")
+assert_frame_equal = pd.testing.assert_frame_equal
+
+SAMPLE_DATA_FOLDER = os.path.join(os.path.dirname(__file__), "sample-data")
+
+TEST_DATA_1 = os.path.join(SAMPLE_DATA_FOLDER, "obs_3day.bufr")
+TEST_DATA_2 = os.path.join(SAMPLE_DATA_FOLDER, "synop_multi_subset_uncompressed.bufr")
+
+# contains 1 message - with 51 compressed subsets with multiple timePeriods
+TEST_DATA_9 = os.path.join(SAMPLE_DATA_FOLDER, "ens_multi_subset_compressed.bufr")
+
+# contains 1 message - with 128 compressed subsets with some str values
+TEST_DATA_10 = os.path.join(SAMPLE_DATA_FOLDER, "pgps_110.bufr")
+
+REF_DATA_FOLDER = os.path.join(os.path.dirname(__file__), "ref_data")
+REF_DATA_1 = os.path.join(REF_DATA_FOLDER, "obs_3day_ref_1.csv")
+REF_DATA_2 = os.path.join(REF_DATA_FOLDER, "synop_uncompressed_ref_1.csv")
+
+
+def test_read_flat_bufr_args() -> None:
+    # The message structure is the same in all the messages
+    # but some have #1#totalPrecipitationPast6Hours while
+    # others have #1#totalPrecipitationPast24Hours at the
+    # same position within the message
+
+    # default args
+    res = pdbufr.read_bufr(TEST_DATA_1, flat=True)
+
+    assert isinstance(res, pd.DataFrame)
+    assert "edition" in res
+    assert "#1#latitude" in res
+    assert "#1#totalPrecipitationPast6Hours" in res
+    assert "#1#totalPrecipitationPast24Hours" in res
+    assert len(res.columns) == 103
+    assert len(res) == 50
+
+    # various positional args
+    for columns in ["", "all"]:
+        res = pdbufr.read_bufr(TEST_DATA_1, columns, flat=True)
+
+        assert isinstance(res, pd.DataFrame)
+        assert "edition" in res
+        assert "#1#latitude" in res
+        assert "#1#totalPrecipitationPast6Hours" in res
+        assert "#1#totalPrecipitationPast24Hours" in res
+        assert len(res.columns) == 103
+        assert len(res) == 50
+
+    # various kw args
+    for columns in [[], "", [""], "all", ["all"]]:  # type: ignore
+        res = pdbufr.read_bufr(TEST_DATA_1, columns=columns, flat=True)
+
+        assert isinstance(res, pd.DataFrame)
+        assert "edition" in res
+        assert "#1#latitude" in res
+        assert "#1#totalPrecipitationPast6Hours" in res
+        assert "#1#totalPrecipitationPast24Hours" in res
+        assert len(res.columns) == 103
+        assert len(res) == 50
+
+    with pytest.raises(ValueError) as exc:
+        res = pdbufr.read_bufr(TEST_DATA_1, "a", flat=True)
+
+    with pytest.raises(ValueError) as exc:
+        res = pdbufr.read_bufr(TEST_DATA_1, ["a", "a"], flat=True)
+
+    with pytest.raises(TypeError) as exc_t:
+        res = pdbufr.read_bufr(TEST_DATA_1, 3, flat=True)  # type: ignore
+
+    with pytest.raises(ValueError) as exc:
+        res = pdbufr.read_bufr(TEST_DATA_1, [3], flat=True)  # type: ignore
+
+    with pytest.raises(ValueError) as exc:
+        res = pdbufr.read_bufr(TEST_DATA_1, [3, 4], flat=True)  # type: ignore
+
+
+def test_read_flat_bufr_one_subset_one_filters() -> None:
+    # The message structure is the same in all the messages
+    # but some have #1#totalPrecipitationPast6Hours while
+    # others have #1#totalPrecipitationPast24Hours at the
+    # same position within the message
+
+    # all
+    res = pdbufr.read_bufr(TEST_DATA_1, "all", flat=True)
+
+    assert isinstance(res, pd.DataFrame)
+    assert "edition" in res
+    assert "#1#latitude" in res
+    assert "#1#totalPrecipitationPast6Hours" in res
+    assert "#1#totalPrecipitationPast24Hours" in res
+    assert len(res.columns) == 103
+    assert len(res) == 50
+
+    # omitting header or data sections
+    res = pdbufr.read_bufr(TEST_DATA_1, "data", flat=True)
+
+    assert isinstance(res, pd.DataFrame)
+    assert "edition" not in res
+    assert "#1#latitude" in res
+    assert len(res.columns) == 53
+    assert len(res) == 50
+
+    res = pdbufr.read_bufr(TEST_DATA_1, "header", flat=True)
+
+    assert isinstance(res, pd.DataFrame)
+    assert "edition" in res
+    assert "#1#latitude" not in res
+    assert len(res.columns) == 50
+    assert len(res) == 50
+
+    # required columns
+    required_columns = ["latitude", ["latitude"], ["latitude", "edition"]]
+    for r in required_columns:
+        res = pdbufr.read_bufr(TEST_DATA_1, "all", flat=True, required_columns=r)
+        print(f"r={r}")
+        assert isinstance(res, pd.DataFrame)
+        assert "edition" in res
+        assert "#1#latitude" in res
+        assert len(res.columns) == 103
+        assert len(res) == 50
+
+    res = pdbufr.read_bufr(
+        TEST_DATA_1, "all", flat=True, required_columns="totalPrecipitationPast6Hours"
+    )
+    assert isinstance(res, pd.DataFrame)
+    assert "edition" in res
+    assert "#1#latitude" in res
+    assert "#1#totalPrecipitationPast6Hours" in res
+    assert len(res.columns) == 102
+    assert len(res) == 43
+
+    res = pdbufr.read_bufr(
+        TEST_DATA_1,
+        "all",
+        flat=True,
+        required_columns="totalPrecipitationPast24Hours",
+    )
+    assert isinstance(res, pd.DataFrame)
+    assert "edition" in res
+    assert "#1#latitude" in res
+    assert "#1#totalPrecipitationPast24Hours" in res
+    assert len(res.columns) == 102
+    assert len(res) == 7
+
+    res = pdbufr.read_bufr(TEST_DATA_1, "all", flat=True, required_columns="xyz")
+
+    assert isinstance(res, pd.DataFrame)
+    assert res.empty
+
+    # omitting header or data sections + required columns
+    required_columns = ["latitude", ["latitude"], ["latitude", "edition"]]
+    for r in required_columns:
+        res = pdbufr.read_bufr(TEST_DATA_1, "data", flat=True, required_columns=r)
+        print(f"header=False r={r}")
+        assert isinstance(res, pd.DataFrame)
+        assert "edition" not in res
+        assert "#1#latitude" in res
+        assert len(res.columns) == 53
+        assert len(res) == 50
+
+    required_columns = ["latitude", ["latitude"], ["latitude", "edition"]]
+    for r in required_columns:
+        res = pdbufr.read_bufr(TEST_DATA_1, "header", flat=True, required_columns=r)
+        print(f"data=False r={r}")
+        assert isinstance(res, pd.DataFrame)
+        assert "edition" in res
+        assert "#1#latitude" not in res
+        assert len(res.columns) == 50
+        assert len(res) == 50
+
+    # filters
+    res = pdbufr.read_bufr(
+        TEST_DATA_1, "all", flat=True, filters={"rdbtimeTime": "115557"}
+    )
+
+    assert isinstance(res, pd.DataFrame)
+    assert "edition" in res
+    assert "#1#latitude" in res
+    assert len(res.columns) == 103
+    assert len(res) == 6
+
+    res = pdbufr.read_bufr(
+        TEST_DATA_1,
+        "all",
+        flat=True,
+        filters={"rdbtimeTime": "115557"},
+        required_columns=["latitude", "edition"],
+    )
+
+    assert isinstance(res, pd.DataFrame)
+    assert "edition" in res
+    assert "#1#latitude" in res
+    assert len(res.columns) == 103
+    assert len(res) == 6
+
+    res = pdbufr.read_bufr(TEST_DATA_1, "all", flat=True, filters={"count": 1})
+
+    assert isinstance(res, pd.DataFrame)
+    assert "edition" in res
+    assert "#1#latitude" in res
+    assert len(res.columns) == 102
+    assert len(res) == 1
+
+    res = pdbufr.read_bufr(
+        TEST_DATA_1, "all", flat=True, filters={"stationNumber": 894}
+    )
+
+    assert isinstance(res, pd.DataFrame)
+    assert "edition" in res
+    assert "#1#latitude" in res
+    assert "#1#totalPrecipitationPast6Hours" in res
+    assert "#1#totalPrecipitationPast24Hours" not in res
+    assert len(res.columns) == 102
+    assert len(res) == 1
+
+    res = pdbufr.read_bufr(
+        TEST_DATA_1, "all", flat=True, filters={"stationNumber": [894, 103]}
+    )
+
+    assert isinstance(res, pd.DataFrame)
+    assert "edition" in res
+    assert "#1#latitude" in res
+    assert "#1#totalPrecipitationPast6Hours" in res
+    assert "#1#totalPrecipitationPast24Hours" in res
+    assert len(res.columns) == 103
+    assert len(res) == 2
+
+    res = pdbufr.read_bufr(
+        TEST_DATA_1, "all", flat=True, filters={"WMO_station_id": [3894, 7103]}
+    )
+
+    assert isinstance(res, pd.DataFrame)
+    assert "edition" in res
+    assert "#1#latitude" in res
+    assert "#1#totalPrecipitationPast6Hours" in res
+    assert "#1#totalPrecipitationPast24Hours" in res
+    assert len(res.columns) == 103
+    assert len(res) == 2
+
+    # compare to csv
+    res = pdbufr.read_bufr(TEST_DATA_1, "all", flat=True, filters={"count": 2})
+
+    assert isinstance(res, pd.DataFrame)
+    assert "edition" in res
+    assert "#1#latitude" in res
+    assert "#1#totalPrecipitationPast6Hours" in res
+    assert "#1#totalPrecipitationPast24Hours" not in res
+    assert len(res.columns) == 102
+    assert len(res) == 1
+
+    # res.to_csv(REF_DATA_1, index=False)
+    ref = pd.read_csv(
+        REF_DATA_1,
+        dtype={"typicalDate": str, "typicalTime": str, "rdbtimeTime": str},
+    )
+
+    assert res.columns.to_list() == ref.columns.to_list()
+    assert_frame_equal(res.iloc[:, :39], ref.iloc[:, :39])
+
+
+def test_read_flat_bufr_uncompressed_subsets() -> None:
+    res = pdbufr.read_bufr(
+        TEST_DATA_2,
+        "all",
+        flat=True,
+    )
+
+    assert isinstance(res, pd.DataFrame)
+    assert "edition" in res
+    assert "#1#latitude" in res
+    assert len(res.columns) == 101
+    assert len(res) == 12
+
+    # required columns
+    res = pdbufr.read_bufr(
+        TEST_DATA_2, "all", flat=True, required_columns=["airTemperature"]
+    )
+
+    assert isinstance(res, pd.DataFrame)
+    assert "edition" in res
+    assert "#1#latitude" in res
+    assert len(res.columns) == 101
+    assert len(res) == 12
+
+    res = pdbufr.read_bufr(
+        TEST_DATA_2, "data", flat=True, required_columns=["airTemperature"]
+    )
+
+    assert isinstance(res, pd.DataFrame)
+    assert "edition" not in res
+    assert "#1#latitude" in res
+    assert len(res.columns) == 80
+    assert len(res) == 12
+
+    res = pdbufr.read_bufr(
+        TEST_DATA_2, "data", flat=True, required_columns=["edition", "airTemperature"]
+    )
+
+    assert isinstance(res, pd.DataFrame)
+    assert "edition" not in res
+    assert "#1#latitude" in res
+    assert len(res.columns) == 80
+    assert len(res) == 12
+
+    res = pdbufr.read_bufr(
+        TEST_DATA_2, "data", flat=True, required_columns=["xyz", "airTemperature"]
+    )
+
+    assert isinstance(res, pd.DataFrame)
+    assert res.empty
+
+    res = pdbufr.read_bufr(
+        TEST_DATA_2,
+        "header",
+        flat=True,
+        required_columns=["edition", "airTemperature"],
+    )
+
+    assert isinstance(res, pd.DataFrame)
+    assert "edition" in res
+    assert "#1#latitude" not in res
+    assert len(res.columns) == 21
+    assert len(res) == 12
+
+    # header filter
+    res = pdbufr.read_bufr(TEST_DATA_2, "all", flat=True, filters={"observedData": 1})
+
+    assert isinstance(res, pd.DataFrame)
+    assert "edition" in res
+    assert "#1#latitude" in res
+    assert len(res.columns) == 101
+    assert len(res) == 12
+
+    # data filter
+    res = pdbufr.read_bufr(TEST_DATA_2, "all", flat=True, filters={"stationNumber": 27})
+
+    assert isinstance(res, pd.DataFrame)
+    assert "edition" in res
+    assert "#1#latitude" in res
+    assert len(res.columns) == 101
+    assert len(res) == 1
+
+    res = pdbufr.read_bufr(
+        TEST_DATA_2, "all", flat=True, filters={"stationNumber": [27, 84]}
+    )
+
+    assert isinstance(res, pd.DataFrame)
+    assert "edition" in res
+    assert "#1#latitude" in res
+    assert "#1#airTemperature" in res
+    assert len(res.columns) == 101
+    assert len(res) == 2
+    ref_val = [276.45, 266.55]
+    np.testing.assert_allclose(ref_val, res["#1#airTemperature"])
+
+    # header + data filter
+    res = pdbufr.read_bufr(
+        TEST_DATA_2,
+        "all",
+        flat=True,
+        filters={"observedData": 1, "stationNumber": [27, 84]},
+    )
+
+    assert isinstance(res, pd.DataFrame)
+    assert "edition" in res
+    assert "#1#latitude" in res
+    assert "#1#airTemperature" in res
+    assert len(res.columns) == 101
+    assert len(res) == 2
+    ref_val = [276.45, 266.55]
+    np.testing.assert_allclose(ref_val, res["#1#airTemperature"])
+
+    # combing all options to together
+    res = pdbufr.read_bufr(
+        TEST_DATA_2,
+        "data",
+        flat=True,
+        filters={"observedData": 1, "stationNumber": [27, 84]},
+        required_columns=["edition", "airTemperature"],
+    )
+
+    assert isinstance(res, pd.DataFrame)
+    assert "edition" not in res
+    assert "#1#latitude" in res
+    assert "#1#airTemperature" in res
+    assert len(res.columns) == 80
+    assert len(res) == 2
+    ref_val = [276.45, 266.55]
+    np.testing.assert_allclose(ref_val, res["#1#airTemperature"])
+
+    res = pdbufr.read_bufr(
+        TEST_DATA_2,
+        "header",
+        flat=True,
+        filters={"observedData": 1, "stationNumber": [27, 84]},
+        required_columns=["edition", "airTemperature"],
+    )
+
+    assert isinstance(res, pd.DataFrame)
+    assert "edition" in res
+    assert "#1#latitude" not in res
+    assert "#1#airTemperature" not in res
+    assert len(res.columns) == 21
+    assert len(res) == 2
+
+    # compare to csv
+    res = pdbufr.read_bufr(
+        TEST_DATA_2,
+        "all",
+        flat=True,
+        filters={"stationNumber": [27, 84]},
+    )
+
+    assert isinstance(res, pd.DataFrame)
+    assert "edition" in res
+    assert "#1#latitude" in res
+    assert "#1#airTemperature" in res
+    assert len(res.columns) == 101
+    assert len(res) == 2
+
+    # res.to_csv("REF_DATA_2", index=False)
+    ref = pd.read_csv(
+        REF_DATA_2,
+        dtype={
+            "typicalDate": str,
+            "typicalTime": str,
+            "#1#heightOfBarometerAboveMeanSeaLevel": str,
+            "#1#nonCoordinatePressure": str,
+            "#1#pressureReducedToMeanSeaLevel": str,
+            "#1#3HourPressureChange": str,
+            "#1#characteristicOfPressureTendency": str,
+            "#1#24HourPressureChange": str,
+        },
+    )
+
+    assert res.columns.to_list() == ref.columns.to_list()
+    assert_frame_equal(res.iloc[:, :39], ref.iloc[:, :39])
+
+
+def test_read_flat_bufr_compressed_subsets() -> None:
+    res = pdbufr.read_bufr(TEST_DATA_9, "all", flat=True)
+
+    ref_val: T.Any
+
+    assert isinstance(res, pd.DataFrame)
+    assert "edition" in res
+    assert "#1#latitude" in res
+    for i in range(1, 62):
+        assert f"#{i}#timePeriod" in res
+        assert f"#{i}#cape" in res
+    assert len(res.columns) == 149
+    assert len(res) == 51
+
+    # required columns
+    res = pdbufr.read_bufr(TEST_DATA_9, "all", flat=True, required_columns=["cape"])
+
+    assert isinstance(res, pd.DataFrame)
+    assert "edition" in res
+    assert "#1#latitude" in res
+    for i in range(1, 62):
+        assert f"#{i}#timePeriod" in res
+        assert f"#{i}#cape" in res
+    assert len(res.columns) == 149
+    assert len(res) == 51
+
+    res = pdbufr.read_bufr(
+        TEST_DATA_9, "data", flat=True, required_columns=["edition", "cape"]
+    )
+
+    assert isinstance(res, pd.DataFrame)
+    assert "edition" not in res
+    assert "#1#latitude" in res
+    for i in range(1, 62):
+        print(i)
+        assert f"#{i}#timePeriod" in res
+        assert f"#{i}#cape" in res
+    assert len(res.columns) == 130
+    assert len(res) == 51
+
+    res = pdbufr.read_bufr(
+        TEST_DATA_9, "data", flat=True, required_columns=["xyz", "cape"]
+    )
+
+    assert isinstance(res, pd.DataFrame)
+    assert res.empty
+
+    res = pdbufr.read_bufr(
+        TEST_DATA_9, "header", flat=True, required_columns=["edition", "cape"]
+    )
+
+    assert isinstance(res, pd.DataFrame)
+    assert "edition" in res
+    assert "#1#latitude" not in res
+    assert len(res.columns) == 19
+    assert len(res) == 51
+
+    # header filter
+    res = pdbufr.read_bufr(TEST_DATA_9, "all", flat=True, filters={"observedData": 1})
+
+    assert isinstance(res, pd.DataFrame)
+    assert "edition" in res
+    assert "#1#latitude" in res
+    assert len(res.columns) == 149
+    assert len(res) == 51
+    ref_val = [1] * 51
+    np.testing.assert_allclose(ref_val, res["observedData"])
+
+    # data filter
+    res = pdbufr.read_bufr(
+        TEST_DATA_9, "all", flat=True, filters={"ensembleMemberNumber": 2}
+    )
+
+    assert isinstance(res, pd.DataFrame)
+    assert "edition" in res
+    assert "#1#latitude" in res
+    assert len(res.columns) == 149
+    assert len(res) == 1
+    ref_val = [2]
+    np.testing.assert_allclose(ref_val, res["#1#ensembleMemberNumber"])
+    ref_val = [174.2]
+    np.testing.assert_allclose(ref_val, res["#2#cape"])
+
+    res = pdbufr.read_bufr(
+        TEST_DATA_9, "all", flat=True, filters={"ensembleMemberNumber": [2, 4]}
+    )
+
+    assert isinstance(res, pd.DataFrame)
+    assert "edition" in res
+    assert "#1#latitude" in res
+    assert len(res.columns) == 149
+    assert len(res) == 2
+    ref_val = [2, 4]
+    np.testing.assert_allclose(ref_val, res["#1#ensembleMemberNumber"])
+    ref_val = [174.2, 200.0]
+    np.testing.assert_allclose(ref_val, res["#2#cape"])
+
+    # header + data filter
+    res = pdbufr.read_bufr(
+        TEST_DATA_9,
+        "all",
+        flat=True,
+        filters={"observedData": 1, "ensembleMemberNumber": [2, 4]},
+    )
+
+    assert isinstance(res, pd.DataFrame)
+    assert "edition" in res
+    assert "#1#latitude" in res
+    assert "#1#cape" in res
+    assert len(res.columns) == 149
+    assert len(res) == 2
+    ref_val = [2, 4]
+    np.testing.assert_allclose(ref_val, res["#1#ensembleMemberNumber"])
+    ref_val = [174.2, 200.0]
+    np.testing.assert_allclose(ref_val, res["#2#cape"])
+
+    # combing all options to together
+    res = pdbufr.read_bufr(
+        TEST_DATA_9,
+        "data",
+        flat=True,
+        filters={"observedData": 1, "ensembleMemberNumber": [2, 4]},
+        required_columns=["edition", "cape"],
+    )
+
+    assert isinstance(res, pd.DataFrame)
+    assert "edition" not in res
+    assert "#1#latitude" in res
+    assert "#1#ensembleMemberNumber" in res
+    assert "#1#cape" in res
+    assert len(res.columns) == 130
+    assert len(res) == 2
+    ref_val = [2, 4]
+    np.testing.assert_allclose(ref_val, res["#1#ensembleMemberNumber"])
+    ref_val = [174.2, 200.0]
+    np.testing.assert_allclose(ref_val, res["#2#cape"])
+
+    res = pdbufr.read_bufr(
+        TEST_DATA_9,
+        "header",
+        flat=True,
+        filters={"observedData": 1, "ensembleMemberNumber": [2, 4]},
+        required_columns=["edition", "cape"],
+    )
+
+    assert isinstance(res, pd.DataFrame)
+    assert "edition" in res
+    assert "#1#latitude" not in res
+    assert "#1#ensembleMemberNumber" not in res
+    assert "#1#cape" not in res
+    assert len(res.columns) == 19
+    assert len(res) == 2
+
+
+def test_read_flat_bufr_compressed_subsets_with_str() -> None:
+    ref_str: T.Any
+
+    res = pdbufr.read_bufr(TEST_DATA_10, "all", flat=True)
+
+    assert isinstance(res, pd.DataFrame)
+    assert "edition" in res
+    assert "#1#latitude" in res
+    assert "#1#stationOrSiteName" in res
+    assert len(res["#1#stationOrSiteName"]) == 128
+    assert isinstance(res["#1#stationOrSiteName"][0], str)
+    ref_str = ["ARD2-LPTR"] * 11 + ["DAV2-LPTR"] * 11
+    assert list(res["#1#stationOrSiteName"][:22]) == ref_str
+    assert len(res.columns) == 228
+    assert len(res) == 128
+
+    res = pdbufr.read_bufr(
+        TEST_DATA_10, filters={"stationOrSiteName": "DAV2-LPTR"}, flat=True
+    )
+
+    assert isinstance(res, pd.DataFrame)
+    assert "edition" in res
+    assert "#1#latitude" in res
+    assert "#1#stationOrSiteName" in res
+    assert len(res["#1#stationOrSiteName"]) == 11
+    assert isinstance(res["#1#stationOrSiteName"][0], str)
+    ref_str = ["DAV2-LPTR"] * 11
+    assert list(res["#1#stationOrSiteName"]) == ref_str
+    assert len(res.columns) == 228
+    assert len(res) == 11
+
+
+def test_read_flat_bufr_warning() -> None:
+    def _find_warning(w: T.Any) -> bool:
+        for item in w:
+            if issubclass(
+                item.category, UserWarning
+            ) and "not all BUFR messages/subsets have the same structure" in str(
+                item.message
+            ):
+                return True
+        return False
+
+    # non-overlapping messages: warning generated
+    with warnings.catch_warnings(record=True) as w:
+        res = pdbufr.read_bufr(TEST_DATA_1, flat=True)
+        assert len(res.columns) == 103
+        assert len(res) == 50
+        assert len(w) > 0
+        assert _find_warning(w)
+
+    # non-overlapping messages: warning disabled
+    warnings.filterwarnings("ignore", module="pdbufr")
+    with warnings.catch_warnings(record=True) as w:
+        res = pdbufr.read_bufr(TEST_DATA_1, flat=True)
+        assert len(res.columns) == 103
+        assert len(res) == 50
+        assert not _find_warning(w)
+
+    # re-enables warnings
+    warnings.filterwarnings("always", module="pdbufr")
+    with warnings.catch_warnings(record=True) as w:
+        res = pdbufr.read_bufr(TEST_DATA_1, flat=True)
+        assert len(res.columns) == 103
+        assert len(res) == 50
+        assert len(w) > 0
+        assert _find_warning(w)
+
+    # overlapping messages: no warnings should be generated
+    with warnings.catch_warnings(record=True) as w:
+        res = pdbufr.read_bufr(TEST_DATA_2, flat=True)
+        assert len(res.columns) == 101
+        assert len(res) == 12
+        assert not _find_warning(w)