Repository: IAMconsortium/pyam
Pull request #315 (2019-12-31 15:26:30)

clean-up of aggregation features
# Please confirm that this PR has done the following:

- [x] Tests Added
- [x] Documentation Added
- [x] Description in RELEASE_NOTES.md Added

# Description of PR

This PR implements a number of clean-ups following PRs #305 and #312:
 - the return type of `aggregate()` and `aggregate_region()` is changed to an `IamDataFrame` instance (per suggestion by @jkikstra), previously a timeseries-dataframe
 - the return type of `check_aggregate()` and `check_aggregate_region()` is changed to a `pd.DataFrame` with both expected and actual value (previously only the expected value)
 - adds an `equals()` function (originally used to make tests easier)
 - adds an `pyam.testing.assert_frames_equal()` function (to make tests easier)
 - the tutorial for aggregation, downscaling and consistency checking is reworked (now includes description of regional components, weighted average, setting variables as list, downscaling)
 - the aggregation tests are completely refactored (because of the changed return types, it was easier to completely rewrite rather than figure out how to salvage)

danielhuppmann: @francescolovat, @fabiosferra, @vruijven, @volker-krey, maybe you have a minute to read through the completely reworked tutorial `aggregating_downscaling_consistency` - in particular whether the behaviour of the new `downscale_region()` function is a good starting point for upcoming work?
danielhuppmann: ping @gidden @znicholls @jkikstra @francescolovat @fabiosferra

I'd like to make a new release (including the new tutorial) in time for the openmod meeting [1], so I'd appreciate your comments by the end of this week. If there are no objections, I'll merge on Jan 13.

[1] https://wiki.openmod-initiative.org/wiki/Open_Energy_Modelling_Workshop_-_Berlin_2020
danielhuppmann: thanks @gidden for the useful comments!
 - added a new testing-module with a function `pyam.testing.assert_frame_equal()`
 - refactored the function names in the `aggregate` module
 - added a longer docstring (copying [pandas.DataFrame.equals](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.equals.html))

Minor version will be bumped in the new release

danielhuppmann: Thanks @jkikstra for your review and comments!

- re the keyword `components` (which is used in both `aggregate()` or `aggregate_region()`): I agree it's not great except that every alternative I can think of is worse... Also, these functions have been implemented in this way for at least a year, so any change should be a real improvement.

 - re `check_internal_consistency()`: the main reason why I didn't change that is laziness to think through what a good return type would be and write enough tests to make sure that there are no weird results in any corner cases. Maybe tackle that in a subsequent PR (cc @znicholls)?
danielhuppmann: Follow-up to @jkikstra's comment on return type:

One option could be something like

```
> pd.concat(tutorial_df.check_internal_consistency().values(), sort=False)
```
<img width="597" alt="image" src="https://user-images.githubusercontent.com/16931589/72045492-782eef80-32b6-11ea-9dc5-faa898006f57.png">

jkikstra: Hi @danielhuppmann Thanks for implementing our suggestions. 

in my eyes your suggestion for the output of `check_internal_consistency()` is a definite improvement, so I would suggest implementing that and then I have no further suggested improvements
gidden: @danielhuppmann I share @znicholls opinion on the test refactor here (also not blocking but with a strong preference).

I may not grok the gritty details, but is it possible to write a wrapper function in the tests that would translate the newly returned `IamDataFrame` back to a `pd.DataFrame` in the test's expected format and then apply that in each test a la `pdt.assert_frame_equals(pd_to_pyam_test(obs), exp)`?
danielhuppmann: re the comments by @znicholls and @gidden:

0) sorry that this is such a big refactoring - I divided it into a suite of PRs to make it manageable, but it's still huge...

1) API of `check_internal_consistency()`: I added a kwarg `components` so that the behaviour can be controlled and passed through to `check_aggregate_region()`. However, as to whether we should use `True` or `False` as default, detecting components that exist only at the region-level is one use case out of many (it's relevant mainly in the emissions domain). And it would interfere with using (weighted) average or other methods which are equally valid and relevant, so I changed that to `False` with a  purpose.

2) about the tests: I agree that just dropping tests is a really bad idea, but the suite of tests were written so efficiently (a dozen different tests with hardly any documentation about their intent, all feeding into one master testing function with a bunch of if-else-logic to do the comparison), it just wasn't possible for me to disentangle this.
@gidden, your approach sounds smart but if you look at what was actually tested (length of indices, etc.) this won't work. The new test suite with `assert_frame_equal()` is far more explicit.
@znicholls, if you can spell out for each of the removed tests what their intent was, I'll be happy to re-introduce them in a next step.
francescolovat: Hi @danielhuppmann,

I've been through the `pyam_first_steps.ipynb` tutorial again. 

It really looks nice. You've implemented all our suggestions in a very detailed way. I was really enjoyable to follow the steps in the notebook.

I have no further comments to add to it (I'll keep in mind some formatting features you included, e.g. the blue boxes, to potentially include them also in the `MESSAGEix` tutorials).

I regards of the previous conversation about the refactoring of tests, I have little experience with them. I'm still getting familiar with `pandas.util.testing` methods. Matt's suggestions of a wrapper function looks promising, however it is also a matter of the time left before the _openmod_ meeting.


znicholls: > 1. sorry that this is such a big refactoring - I divided it into a suite of PRs to make it manageable, but it's still huge...

All good there's always a nasty conflict between time pressure and moving slowly enough that everyone can keep up (you're striking a good balance)

> 2\. API of `check_internal_consistency()`

All makes sense thanks for spelling it out.

> 3\. about the tests

fair, they were definitely not as clear as they should have been (I have since learnt that whilst this 'efficiency' approach looks good, it's actually a terrible idea as it's not explicit enough about what is going on). Let's park this in <issue> for now
gidden: Ok all, will merge this now that everything is approved/reviewed/marked for future issues. Thanks @danielhuppmann for the huge effort here and all reviewers!

Patch:
diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md
index baac3b5..ad007c3 100644
--- a/RELEASE_NOTES.md
+++ b/RELEASE_NOTES.md
@@ -6,8 +6,18 @@ PR [#305](https://github.com/IAMconsortium/pyam/pull/305) changed the default
 behaviour of `aggregate_region()` regarding the treatment of components at the
 region-level. To keep the previous behaviour, add `components=True`.
 
+PR [#315](https://github.com/IAMconsortium/pyam/pull/314) changed the return
+type of `aggregate[_region]()` to an `IamDataFrame` instance. 
+To keep the previous behaviour, add `timeseries()`.
+The object returned by `[check_]aggregate[_region]()` now includes both the
+actual and the expected value as a `pd.DataFrame` instance.
+The function `check_internal_consistency()` now returns a concatenated dataframe
+rather than a dictionary and also includes optional treatment of components
+(see paragraph above). To keep the previous behaviour, add `components=True`.
+
 ## Individual Updates
 
+- [#315](https://github.com/IAMconsortium/pyam/pull/315) Add `equals()` feature, change return types of `[check_]aggregate[_region]()`, rework aggregation tutorial
 - [#314](https://github.com/IAMconsortium/pyam/pull/314) Update IPCC color scheme colors and add SSP-only colors
 - [#313](https://github.com/IAMconsortium/pyam/pull/313) Add feature to `downscale` timeseries data to subregions using another variable as proxy
 - [#312](https://github.com/IAMconsortium/pyam/pull/312) Allow passing list of variables to `aggregate` functions
diff --git a/doc/source/tutorials.rst b/doc/source/tutorials.rst
index d692fc1..3049b63 100644
--- a/doc/source/tutorials.rst
+++ b/doc/source/tutorials.rst
@@ -8,8 +8,8 @@ Tutorials
    :maxdepth: 1
 
    tutorials/pyam_first_steps.ipynb
-   tutorials/aggregating_variables_and_plotting_with_negative_values.ipynb
+   tutorials/aggregating_downscaling_consistency.ipynb.ipynb
    tutorials/ipcc_colors.ipynb
-   tutorials/pyam_logo.ipynb
    tutorials/iiasa_dbs.ipynb
-   tutorials/checking_databases.ipynb
+   tutorials/aggregating_variables_and_plotting_with_negative_values.ipynb
+   tutorials/pyam_logo.ipynb
diff --git a/doc/source/tutorials/aggregating_downscaling_consistency.ipynb b/doc/source/tutorials/aggregating_downscaling_consistency.ipynb
new file mode 100644
index 0000000..968bc3f
--- /dev/null
+++ b/doc/source/tutorials/aggregating_downscaling_consistency.ipynb
@@ -0,0 +1,445 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Aggregating and downscaling timeseries data\n",
+    "\n",
+    "The **pyam** package offers many tools to facilitate processing of scenario data.\n",
+    "In this notebook, we illustrate methods to aggregate and downscale timeseries data of an `IamDataFrame` across regions and sectors, as well as checking consistency of given data along these dimensions.\n",
+    "\n",
+    "In this tutorial, we show how to make the most of **pyam** to compute such aggregate timeseries data, and to check that a scenario ensemble (or just a single scenario) is complete and that timeseries data \"add up\" across regions and along the variable tree (i.e., that the sum of values of the subcategories such as `Primary Energy|*` are identical to the values of the category `Primary Energy`).\n",
+    "\n",
+    "There are two distinct use cases where these features can be used.\n",
+    "\n",
+    "### Use case 1: compute data at higher/lower sectoral or spatial aggregation\n",
+    "\n",
+    "Given scenario results at a specific (usually very detailed) sectoral and spatial resolution, **pyam** offers a suite of functions to easily compute aggregate timeseries. For example, this allows to sum up national energy demand to regional or global values,\n",
+    "or to compute the average of a global carbon price weighted by regional emissions.\n",
+    "\n",
+    "These functions can be used as part of an automated workflow to generate complete scenario results from raw model outputs.\n",
+    "\n",
+    "### Use case 2: check the consistency of data across sectoral or spatial levels\n",
+    "\n",
+    "In model comparison exercises or ensemble compilation projects, a user needs to verify the internal consistency of submitted scenario results (cf. Huppmann et al., 2018, doi: [10.1038/s41558-018-0317-4](http://rdcu.be/9i8a)).\n",
+    "Such inconsistencies can be due to incomplete variable hierarchies, reporting templates incompatible with model specifications, or user error.\n",
+    "\n",
+    "## Overview\n",
+    "\n",
+    "This notebook illustrates the following features:\n",
+    "\n",
+    "0. Load timeseries data from a snapshot file and inspect the scenario\n",
+    "1. Aggregate timeseries over sectors (i.e., sub-categories)\n",
+    "2. Aggregate timeseries over regions including weighted average\n",
+    "3. Downscale timeseries given at a region level to sub-regions using a proxy variable\n",
+    "4. Check the internal consistency of a scenario (ensemble)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "import pyam"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 0. Load timeseries data from snapshot file and inspect the scenario\n",
+    "\n",
+    "The stylized scenario used in this tutorial has data for two regions (`reg_a` & `reg_b`) as well as the `World` aggregate, and for categories of variables: primary energy demand, emissions, carbon price, and population."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df = pyam.IamDataFrame(data='tutorial_data_aggregating_downscaling.csv')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df.regions()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df.variables()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. Aggregating timeseries across sectors\n",
+    "\n",
+    "Let's first display the data for the components of primary energy demand."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df.filter(variable='Primary Energy|*').timeseries()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Next, we are going to use the [aggregate()](https://pyam-iamc.readthedocs.io/en/stable/api.html#pyam.IamDataFrame.aggregate) function to compute the total `Primary Energy` from its components (wind and coal) in each region (including `World`).\n",
+    "\n",
+    "The function returns an `IamDataFrame`, so we can use [timeseries()](https://pyam-iamc.readthedocs.io/en/stable/api.html#pyam.IamDataFrame.timeseries) to display the resulting data."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df.aggregate('Primary Energy').timeseries()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "If we are interested in **use case 1**, we could use the argument `append=True` to directly add the computed aggregate to the `IamDataFrame`.\n",
+    "\n",
+    "However, in this tutorial, the data already includes the total primary energy demand. Therefore, we illustrate **use case 2** and apply the [check_aggregate()](https://pyam-iamc.readthedocs.io/en/stable/api.html#pyam.IamDataFrame.check_aggregate) function to  verify whether a given variable is the sum of its sectoral components\n",
+    "(i.e., `Primary Energy` should be equal to `Primary Energy|Coal` plus `Primary Energy|Wind`).\n",
+    "The validation is performed separately for each region.\n",
+    "\n",
+    "The function returns `None` if the validation is correct (which it is for primary energy demand)\n",
+    "or a [pandas.DataFrame](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html) highlighting where the aggregate does not match (this will be illustrated in the next section)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df.check_aggregate('Primary Energy')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The function also returns useful logging messages when there is nothing to check (because there are no sectors below `Primary Energy|Wind`)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df.check_aggregate('Primary Energy|Wind')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. Aggregating timeseries across subregions\n",
+    "\n",
+    "Similarly to the previous example, we now use the [aggregate_region()](https://pyam-iamc.readthedocs.io/en/stable/api.html#pyam.IamDataFrame.aggregate_region) function to compute regional aggregates.\n",
+    "By default, this method sums all the regions in the dataframe to make a `World` region; this can be changed with the keyword arguments `region` and `subregions`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df.aggregate_region('Primary Energy').timeseries()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Adding regional components\n",
+    "\n",
+    "As a next step, we use [check_aggregate_region()](https://pyam-iamc.readthedocs.io/en/stable/api.html#pyam.IamDataFrame.check_aggregate_region) to verify that the regional aggregate of CO2 emissions matches the timeseries data given in the scenario."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df.check_aggregate_region('Emissions|CO2')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "As announced above, this validation failed and we see a dataframe of the expected data at the `region` level and the aggregation computed from the `subregions`.\n",
+    "\n",
+    "Let's look at the entire emissions timeseries in the scenario to find out what is going on."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df.filter(variable='Emissions*').timeseries()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Investigating the data carefully, you will notice that emissions from the energy sector and agriculture, forestry & land use (AFOLU) are given in the subregions and the `World` region, whereas emissions from bunker fuels are only defined at the global level.\n",
+    "This is a common issue in emissions data, where some sources (e.g., global aviation and maritime transport) cannot be attributed to one region.\n",
+    "\n",
+    "Luckily, the functions [aggregate_region()](https://pyam-iamc.readthedocs.io/en/stable/api.html#pyam.IamDataFrame.aggregate_region)\n",
+    "and [check_aggregate_region()](https://pyam-iamc.readthedocs.io/en/stable/api.html#pyam.IamDataFrame.check_aggregate_region)\n",
+    "support this use case:\n",
+    "by adding `components=True`, the regional aggregation will include any sub-categories of the variable that are only present at the `region` level but not in any subregion."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df.aggregate_region('Emissions|CO2', components=True).timeseries()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The regional aggregate now matches the data given at the `World` level in the tutorial data.\n",
+    "\n",
+    "Note that the components to be included at the region level can also be specified directly via a list of variables, in this case we would use `components=['Emissions|CO2|Bunkers']`."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Computing a weighted average across regions\n",
+    "\n",
+    "One other frequent requirement when aggregating across regions is a weighted average.\n",
+    "\n",
+    "To illustrate this feature, the tutorial data includes carbon price data.\n",
+    "Naturally, the appropriate weighting data are the regional carbon emissions.\n",
+    "\n",
+    "The following cells show:\n",
+    "\n",
+    "0. The carbon price data across the regions\n",
+    "1. A (failing) validation that the regional aggretion (without weights) matches the reported prices at the `World` level\n",
+    "2. The emissions-weighted average of carbon prices returned as a new `IamDataFrame`"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df.filter(variable='Price|Carbon').timeseries()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df.check_aggregate_region('Price|Carbon')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df.aggregate_region('Price|Carbon', weight='Emissions|CO2').timeseries()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3. Downscaling timeseries data to subregions\n",
+    "\n",
+    "The inverse operation of regional aggregation is \"downscaling\" of timeseries data given at a regional level to a number of subregions, usually using some other data as proxy to divide and allocate the total to the subregions.\n",
+    "\n",
+    "This section shows an example using the [downscale_region()](https://pyam-iamc.readthedocs.io/en/stable/api.html#pyam.IamDataFrame.downscale_region) function to divide the total primary energy demand using population as a proxy."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df.downscale_region('Primary Energy', proxy='Population').timeseries()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "By the way, the functions\n",
+    "[aggregate()](https://pyam-iamc.readthedocs.io/en/stable/api.html#pyam.IamDataFrame.aggregate), \n",
+    "[aggregate_region()](https://pyam-iamc.readthedocs.io/en/stable/api.html#pyam.IamDataFrame.aggregate_region) and\n",
+    "[downscale_region()](https://pyam-iamc.readthedocs.io/en/stable/api.html#pyam.IamDataFrame.downscale_region)\n",
+    "also take lists of variables as `variable` argument.\n",
+    "See the next cell for an example."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "var_list = ['Primary Energy', 'Primary Energy|Coal']\n",
+    "df.downscale_region(var_list, proxy='Population').timeseries()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 4. Checking complete internal consistency of a scenario (ensemble)\n",
+    "\n",
+    "The previous sections illustrated two functions to validate specific variables across their sectors (sub-categories) or regional disaggregation.\n",
+    "These two functions are combined in the [check_internal_consistency()](https://pyam-iamc.readthedocs.io/en/stable/api.html#pyam.IamDataFrame.check_internal_consistency) feature.\n",
+    "\n",
+    "<div class=\"alert alert-block alert-warning\">\n",
+    "    This feature of the <b>pyam</b> package currently only supports \"consistency\"\n",
+    "    in the sense of a strictly hierarchical variable tree\n",
+    "    (with subcategories summing up to the category value including components, discussed above)\n",
+    "    and that all the regions sum to the <b>World</b> region.\n",
+    "    See <a href=\"https://github.com/IAMconsortium/pyam/issues/106\">this issue</a> for more information.\n",
+    "</div>\n",
+    "\n",
+    "If we have an internally consistent scenario ensemble (or single scenario), the function will return `None`; otherwise, it will return a concatenation of [pandas.DataFrames](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html) indicating all detected inconsistencies.\n",
+    "\n",
+    "For this section, we use a tutorial scenario which is constructed to highlight the individual validation features below.\n",
+    "The scenario below has two inconsistencies:\n",
+    "\n",
+    "1. In year `2010` and regions `region_b` & `World`, the values of coal and wind do not add up to the total `Primary Energy` value\n",
+    "2. In year `2020` in the `World` region, the value of `Primary Energy` and `Primary Energy|Coal` is not the sum of `region_a` and `region_b` <br />\n",
+    "   (but the sum of wind and coal to `Primary Energy` in each sub-region  is correct)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "tutorial_df = pyam.IamDataFrame(pd.DataFrame([\n",
+    "    ['World', 'Primary Energy', 'EJ/y', 7, 15],\n",
+    "    ['World', 'Primary Energy|Coal', 'EJ/y', 4, 11],\n",
+    "    ['World', 'Primary Energy|Wind', 'EJ/y', 2, 4],\n",
+    "    ['region_a', 'Primary Energy', 'EJ/y', 4, 8],\n",
+    "    ['region_a', 'Primary Energy|Coal', 'EJ/y', 2, 6],\n",
+    "    ['region_a', 'Primary Energy|Wind', 'EJ/y', 2, 2],\n",
+    "    ['region_b', 'Primary Energy', 'EJ/y', 3, 6],\n",
+    "    ['region_b', 'Primary Energy|Coal', 'EJ/y', 2, 4],\n",
+    "    ['region_b', 'Primary Energy|Wind', 'EJ/y', 0, 2],\n",
+    "],\n",
+    "    columns=['region', 'variable', 'unit', 2010, 2020]\n",
+    "), model='model_a', scenario='scen_a')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "All checking-functions take arguments for [np.is_close()](https://docs.scipy.org/doc/numpy/reference/generated/numpy.isclose.html) as keyword arguments. We show our recommended settings and how to use them here."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "np_isclose_args = {\n",
+    "    'equal_nan': True,\n",
+    "    'rtol': 1e-03,\n",
+    "    'atol': 1e-05,\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "tutorial_df.check_internal_consistency(**np_isclose_args)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The output of this function reports both types of illustrative inconsistencies in the scenario constructed for this section.\n",
+    "The log also shows that the other two variables (coal and wind) cannot be assessed because they have no subcategories.\n",
+    "\n",
+    "<div class=\"alert alert-block alert-info\">\n",
+    "    In practice, it would now be up to the user to determine\n",
+    "    the cause of the inconsistency (or confirm that this is expected for some reason).\n",
+    "</div>"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.7.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/doc/source/tutorials/aggregating_variables_and_plotting_with_negative_values.ipynb b/doc/source/tutorials/aggregating_variables_and_plotting_with_negative_values.ipynb
index d6ce245..f3c8724 100644
--- a/doc/source/tutorials/aggregating_variables_and_plotting_with_negative_values.ipynb
+++ b/doc/source/tutorials/aggregating_variables_and_plotting_with_negative_values.ipynb
@@ -235,7 +235,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Even if there are sectors which are defined only at the world level (e.g. `Emissions|CO2|Fossil|Aviation` in our example), Pyam will find them and include them when calculating the regional total."
+    "Even if there are sectors which are defined only at the world level (e.g. `Emissions|CO2|Fossil|Aviation` in our example), Pyam will find them and include them when calculating the regional total if we specify `components=True` when using `aggregate_region`."
    ]
   },
   {
@@ -244,7 +244,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "df.aggregate_region(\"Emissions|CO2|Fossil\")"
+    "df.aggregate_region(\"Emissions|CO2|Fossil\", components=True).timeseries()"
    ]
   },
   {
@@ -292,8 +292,8 @@
    "outputs": [],
    "source": [
     "aviation_emms = df.filter(variable=\"*Aviation*\").timeseries()\n",
-    "aggregate_emms = df.aggregate_region(\"Emissions|CO2|Fossil\")\n",
-    "aggregate_emms_region_only = df.filter(region=\"World\", keep=False).aggregate_region(\"Emissions|CO2|Fossil\")\n",
+    "aggregate_emms = df.aggregate_region(\"Emissions|CO2|Fossil\", components=True).timeseries()\n",
+    "aggregate_emms_region_only = df.filter(region=\"World\", keep=False).aggregate_region(\"Emissions|CO2|Fossil\").timeseries()\n",
     "np.isclose(aggregate_emms.values, aggregate_emms_region_only.values + aviation_emms.values)"
    ]
   }
@@ -314,7 +314,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.7.4"
+   "version": "3.7.6"
   }
  },
  "nbformat": 4,
diff --git a/doc/source/tutorials/checking_consistency.ipynb b/doc/source/tutorials/checking_consistency.ipynb
deleted file mode 100644
index e208796..0000000
--- a/doc/source/tutorials/checking_consistency.ipynb
+++ /dev/null
@@ -1,236 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Checking consistency of a scenario ensemble\n",
-    "\n",
-    "It has happened in previous model comparison exercises that the reported data was not internally consistent. This can be due to incomplete variable hierarchies, reporting templates incompatible with model specifications, or user error.\n",
-    "\n",
-    "In this tutorial, we show how to make the most of **pyam** to check that a scenario ensemble (or just a single scenario) is complete and that timeseries data \"add up\" across regions and along the variable tree (i.e., that the sum of values of the subcategories such as `Primary Energy|*` are identical to the values of the category `Primary Energy`).\n",
-    "\n",
-    "<div class=\"alert alert-block alert-warning\">\n",
-    "    This feature of the <b>pyam</b> package currently only supports \"consistency\"\n",
-    "    in the sense of a strictly hierarchical variable tree\n",
-    "    (with subcategories summing up to the category value)\n",
-    "    and subregions of depth 1 adding up the \"World\" region.\n",
-    "</div>"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import pandas as pd\n",
-    "import pyam"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "We start with a hypothetical tutorial data set, which is constructed to highlight the individual validation features below.\n",
-    "\n",
-    "The scenario below has two inconsistencies:\n",
-    "\n",
-    "1. In year `2010` and regions `region_b` & `World`, the values of coal and wind do not add up to the total `Primary Energy` value\n",
-    "2. In year `2020` in the `World` region, the value of `Primary Energy` and `Primary Energy|Coal` is not the sum of `region_a` and `region_b` <br />\n",
-    "   (but the sum of wind and coal to `Primary Energy` in each sub-region  is correct)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "tutorial_df = pd.DataFrame([\n",
-    "    ['World', 'Primary Energy', 'EJ/y', 7, 15],\n",
-    "    ['World', 'Primary Energy|Coal', 'EJ/y', 4, 11],\n",
-    "    ['World', 'Primary Energy|Wind', 'EJ/y', 2, 4],\n",
-    "    ['region_a', 'Primary Energy', 'EJ/y', 4, 8],\n",
-    "    ['region_a', 'Primary Energy|Coal', 'EJ/y', 2, 6],\n",
-    "    ['region_a', 'Primary Energy|Wind', 'EJ/y', 2, 2],\n",
-    "    ['region_b', 'Primary Energy', 'EJ/y', 3, 6],\n",
-    "    ['region_b', 'Primary Energy|Coal', 'EJ/y', 2, 4],\n",
-    "    ['region_b', 'Primary Energy|Wind', 'EJ/y', 0, 2],\n",
-    "],\n",
-    "    columns=['region', 'variable', 'unit', 2010, 2020]\n",
-    ")\n",
-    "\n",
-    "df = pyam.IamDataFrame(data=tutorial_df, model='model_a', scenario='scen_a')"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Summary\n",
-    "\n",
-    "With the [check_internal_consistency()](https://pyam-iamc.readthedocs.io/en/stable/api.html#pyam.IamDataFrame.check_internal_consistency) feature, we can check the internal consistency of a scenario ensemble (i.e., an `IamDataFrame` instance).\n",
-    "If this method returns `None`, the database is internally consistent (i.e. the total variables are the sum of the sectoral breakdowns and the regional breakdown).\n",
-    "\n",
-    "In the rest of this tutorial, we give you a chance to better understand this method. We go through what it is actually doing and show you the kind of output you can expect."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Checking that variables are the sum of their components\n",
-    "\n",
-    "We are going to use the [check_aggregate()](https://pyam-iamc.readthedocs.io/en/stable/api.html#pyam.IamDataFrame.check_aggregate) method of the `IamDataFrame`\n",
-    "to check that the components of a variable add up to its total.\n",
-    "This method takes [np.is_close()](https://docs.scipy.org/doc/numpy/reference/generated/numpy.isclose.html) arguments as keyword arguments. We show our recommended settings here."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "np_isclose_args = {\n",
-    "    'equal_nan': True,\n",
-    "    'rtol': 1e-03,\n",
-    "    'atol': 1e-05,\n",
-    "}"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "The [check_aggregate()](https://pyam-iamc.readthedocs.io/en/stable/api.html#pyam.IamDataFrame.check_aggregate) function allows us to quickly verify whether a given variable is the sum of its sectoral components (e.g. `Primary Energy` should be equal to `Primary Energy|Coal` plus `Primary Energy|Wind`). The validation is performed separately for each region.\n",
-    "\n",
-    "This section illustrates the first constructed inconsistency in this scenario. The returned [pandas.DataFrame](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html) indicates where the aggregate is not equal to the sum of components."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "df.check_aggregate('Primary Energy', **np_isclose_args)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "In practice, it would now be up to the user to determine the cause of the inconsistency (or confirm that this is expected for some reason)."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### Checking multiple variables\n",
-    "\n",
-    "We can now construct a loop over all variables in this `IamDataFrame`."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "for variable in df.variables():\n",
-    "    df.check_aggregate(variable, **np_isclose_args)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "The log tells us the same message as in the previous example, and it shows that the other two variables (coal and wind) cannot be assessed because they have no subcategories.\n",
-    "\n",
-    "<div class=\"alert alert-block alert-info\">\n",
-    "Note that the detailed output (i.e., where the aggregation validation fails) is not shown in a notebook when calling the function within a loop.<br />\n",
-    "    Read <a href=\"https://www.dataquest.io/blog/jupyter-notebook-tips-tricks-shortcuts/\">this page</a> for helpful tips and tricks when working with Jupyter notebooks.\n",
-    "</div>"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Checking that timeseries subregions sum to aggregate regions\n",
-    "\n",
-    "Similarly to checking that the sum of a variable's components give the declared total shown above, we can check that summing over subregions returns the value of a region.\n",
-    "\n",
-    "To do this, we use the [check_aggregate_region](https://pyam-iamc.readthedocs.io/en/stable/api.html#pyam.IamDataFrame.check_aggregate_region) function. By default, this method checks that all the regions in the dataframe sum to `World`. \n",
-    "\n",
-    "Using this function allows us to quickly check if a regional total for a single variable is equal to the sum of its regional values.\n",
-    "This section illustrates the second constructed inconsistency in this scenario. \n",
-    "The returned [pandas.DataFrame](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html) indicates where the timeseries at the `region='World'` level is not equal to the sum of regional components."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "df.check_aggregate_region('Primary Energy', **np_isclose_args)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Checking complete internal consistency of a scenario (ensemble)\n",
-    "\n",
-    "The previous sections illustrated two functions to validate specific variables across their subcategories or regional breakdown. These two functions are combined in the [check_internal_consistency()](https://pyam-iamc.readthedocs.io/en/stable/api.html#pyam.IamDataFrame.check_internal_consistency) feature.\n",
-    "\n",
-    "If we have an internally consistent scenario ensemble (or single scenario), the function will return `None`; otherwise, it will return a [pandas.DataFrame](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html) indicating all detected inconsistencies.\n",
-    "\n",
-    "<div class=\"alert alert-block alert-warning\">\n",
-    "    Note that at the moment, this method assumes that all the regions sum to the <b>World</b> region. See <a href=\"https://github.com/IAMconsortium/pyam/issues/106\">this issue</a> for more information.\n",
-    "</div>"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "df.check_internal_consistency()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "The output of this function reports both types of illustrative inconsistencies in the scenario constructed for this tutorial."
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.7.4"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
diff --git a/doc/source/tutorials/tutorial_data_aggregating_downscaling.csv b/doc/source/tutorials/tutorial_data_aggregating_downscaling.csv
new file mode 100644
index 0000000..df15bc7
--- /dev/null
+++ b/doc/source/tutorials/tutorial_data_aggregating_downscaling.csv
@@ -0,0 +1,26 @@
+Model,Scenario,Region,Variable,Unit,2005,2010
+model_a,scen_a,World,Emissions|CO2,EJ/y,10.0,14.0
+model_a,scen_a,World,Emissions|CO2|AFOLU,EJ/y,3.0,4.0
+model_a,scen_a,World,Emissions|CO2|Bunkers,EJ/y,1.0,2.0
+model_a,scen_a,World,Emissions|CO2|Energy,EJ/y,6.0,8.0
+model_a,scen_a,World,Population,m,3.0,5.0
+model_a,scen_a,World,Price|Carbon,USD/tCO2,4.0,27.0
+model_a,scen_a,World,Primary Energy,EJ/y,12.0,15.0
+model_a,scen_a,World,Primary Energy|Coal,EJ/y,9.0,10.0
+model_a,scen_a,World,Primary Energy|Wind,EJ/y,3.0,5.0
+model_a,scen_a,reg_a,Emissions|CO2,EJ/y,6.0,8.0
+model_a,scen_a,reg_a,Emissions|CO2|AFOLU,EJ/y,2.0,3.0
+model_a,scen_a,reg_a,Emissions|CO2|Energy,EJ/y,4.0,5.0
+model_a,scen_a,reg_a,Population,m,2.0,3.0
+model_a,scen_a,reg_a,Price|Carbon,USD/tCO2,1.0,30.0
+model_a,scen_a,reg_a,Primary Energy,EJ/y,8.0,9.0
+model_a,scen_a,reg_a,Primary Energy|Coal,EJ/y,6.0,6.0
+model_a,scen_a,reg_a,Primary Energy|Wind,EJ/y,2.0,3.0
+model_a,scen_a,reg_b,Emissions|CO2,EJ/y,3.0,4.0
+model_a,scen_a,reg_b,Emissions|CO2|AFOLU,EJ/y,1.0,1.0
+model_a,scen_a,reg_b,Emissions|CO2|Energy,EJ/y,2.0,3.0
+model_a,scen_a,reg_b,Population,m,1.0,2.0
+model_a,scen_a,reg_b,Price|Carbon,USD/tCO2,10.0,21.0
+model_a,scen_a,reg_b,Primary Energy,EJ/y,4.0,6.0
+model_a,scen_a,reg_b,Primary Energy|Coal,EJ/y,3.0,4.0
+model_a,scen_a,reg_b,Primary Energy|Wind,EJ/y,1.0,2.0
diff --git a/pyam/_aggregate.py b/pyam/_aggregate.py
new file mode 100644
index 0000000..66644cd
--- /dev/null
+++ b/pyam/_aggregate.py
@@ -0,0 +1,155 @@
+import numpy as np
+import logging
+
+from pyam.logging import adjust_log_level
+from pyam.utils import (
+    islistable,
+    isstr,
+    KNOWN_FUNCS,
+    META_IDX,
+)
+
+logger = logging.getLogger(__name__)
+
+
+def _aggregate(df, variable, components=None, method=np.sum):
+    """Internal implementation of the `aggregate` function"""
+    # list of variables require default components (no manual list)
+    if islistable(variable) and components is not None:
+        raise ValueError('aggregating by list of variables cannot use '
+                         'custom components')
+
+    mapping = {}
+    msg = 'cannot aggregate variable `{}` because it has no components'
+    # if single variable
+    if isstr(variable):
+        # default components to all variables one level below `variable`
+        components = components or df._variable_components(variable)
+
+        if not len(components):
+            logger.info(msg.format(variable))
+            return
+
+        for c in components:
+            mapping[c] = variable
+
+    # else, use all variables one level below `variable` as components
+    else:
+        for v in variable if islistable(variable) else [variable]:
+            _components = df._variable_components(v)
+            if not len(_components):
+                logger.info(msg.format(v))
+                continue
+
+            for c in _components:
+                mapping[c] = v
+
+    # rename all components to `variable` and aggregate
+    _df = df.data[df._apply_filters(variable=mapping.keys())].copy()
+    _df['variable'].replace(mapping, inplace=True)
+    return _group_and_agg(_df, [], method)
+
+
+def _aggregate_region(self, variable, region, subregions=None,
+                      components=False, method='sum', weight=None):
+    """Internal implementation for aggregating data over subregions"""
+    if not isstr(variable) and components is not False:
+        msg = 'aggregating by list of variables with components ' \
+              'is not supported'
+        raise ValueError(msg)
+
+    if weight is not None and components is not False:
+        msg = 'using weights and components in one operation not supported'
+        raise ValueError(msg)
+
+    # default subregions to all regions other than `region`
+    subregions = subregions or self._all_other_regions(region, variable)
+
+    if not len(subregions):
+        msg = 'cannot aggregate variable `{}` to `{}` because it does not'\
+              ' exist in any subregion'
+        logger.info(msg.format(variable, region))
+
+        return
+
+    # compute aggregate over all subregions
+    subregion_df = self.filter(region=subregions)
+    rows = subregion_df._apply_filters(variable=variable)
+    if weight is None:
+        col = 'region'
+        _data = _group_and_agg(subregion_df.data[rows], col, method=method)
+    else:
+        weight_rows = subregion_df._apply_filters(variable=weight)
+        _data = _agg_weight(subregion_df.data[rows],
+                            subregion_df.data[weight_rows], method)
+
+    # if not `components=False`, add components at the `region` level
+    if components is not False:
+        with adjust_log_level(logger):
+            region_df = self.filter(region=region)
+
+        # if `True`, auto-detect `components` at the `region` level,
+        # defaults to variables below `variable` only present in `region`
+        if components is True:
+            level = dict(level=None)
+            r_comps = region_df._variable_components(variable, **level)
+            sr_comps = subregion_df._variable_components(variable, **level)
+            components = set(r_comps).difference(sr_comps)
+
+        if len(components):
+            # rename all components to `variable` and aggregate
+            rows = region_df._apply_filters(variable=components)
+            _df = region_df.data[rows].copy()
+            _df['variable'] = variable
+            _data = _data.add(_group_and_agg(_df, 'region'), fill_value=0)
+
+    return _data
+
+
+def _group_and_agg(df, by, method=np.sum):
+    """Groupby & aggregate `df` by column(s), return indexed `pd.Series`"""
+    by = [by] if isstr(by) else by
+    cols = [c for c in list(df.columns) if c not in ['value'] + by]
+    # pick aggregator func (default: sum)
+    return df.groupby(cols)['value'].agg(_get_method_func(method))
+
+
+def _agg_weight(df, weight, method):
+    """Aggregate `df` by regions with weights, return indexed `pd.Series`"""
+    # only summation allowed with weights
+    if method not in ['sum', np.sum]:
+        raise ValueError('only method `np.sum` allowed for weighted average')
+
+    cols = _list_diff(df.columns, ['variable', 'unit', 'value'])
+    _weight = _get_value_col(weight, cols)
+
+    if not _get_value_col(df, cols).index.equals(_weight.index):
+        raise ValueError('inconsistent index between variable and weight')
+
+    _data = _get_value_col(df)
+    col1 = _list_diff(_data.index.names, ['region'])
+    col2 = META_IDX + [i for i in ['year', 'time'] if i in _weight.index.names]
+    return (_data * _weight).groupby(col1).sum() / _weight.groupby(col2).sum()
+
+
+def _list_diff(lst, exclude):
+    """Return the list minus those elements in `exclude`"""
+    return [i for i in lst if i not in exclude]
+
+
+def _get_value_col(df, cols=None):
+    """Return the value column as `pd.Series sorted by index"""
+    cols = cols or [i for i in df.columns if i != 'value']
+    return df.set_index(cols)['value'].sort_index()
+
+
+def _get_method_func(method):
+    """Translate a string to a known method"""
+    if not isstr(method):
+        return method
+
+    if method in KNOWN_FUNCS:
+        return KNOWN_FUNCS[method]
+
+    # raise error if `method` is a string but not in dict of known methods
+    raise ValueError('method `{}` is not a known aggregator'.format(method))
diff --git a/pyam/core.py b/pyam/core.py
index 664a671..0d19009 100755
--- a/pyam/core.py
+++ b/pyam/core.py
@@ -15,8 +15,7 @@ except ImportError:
     has_ix = False
 
 from pyam import plotting
-
-from pyam.logging import adjust_log_level, deprecation_warning
+from pyam.logging import deprecation_warning
 from pyam.run_control import run_control
 from pyam.utils import (
     write_sheet,
@@ -37,11 +36,11 @@ from pyam.utils import (
     META_IDX,
     YEAR_IDX,
     IAMC_IDX,
-    SORT_IDX,
-    KNOWN_FUNCS
+    SORT_IDX
 )
 from pyam.read_ixmp import read_ix
 from pyam.timeseries import fill_series
+from pyam._aggregate import _aggregate, _aggregate_region, _group_and_agg
 
 logger = logging.getLogger(__name__)
 
@@ -169,6 +168,27 @@ class IamDataFrame(object):
         """Indicator whether ``IamDataFrame`` is empty"""
         return self.data.empty
 
+    def equals(self, other):
+        """Test if two objects contain the same data and meta indicators
+
+        This function allows two IamdDataFrame instances to be compared against
+        each other to see if they have the same timeseries data and meta
+        indicators. NaNs in the same location of the meta table are considered
+        equal.
+
+        Parameters
+        ----------
+        other: IamDataFrame
+            The other IamDataFrame to be compared with the first.
+        """
+        if not isinstance(other, IamDataFrame):
+            raise ValueError('`other` is not an `IamDataFrame` instance')
+
+        if compare(self, other).empty and self.meta.equals(other.meta):
+            return True
+        else:
+            return False
+
     def models(self):
         """Get a list of models"""
         return pd.Series(self.meta.index.levels[0])
@@ -783,45 +803,17 @@ class IamDataFrame(object):
             append the aggregate timeseries to `self` and return None,
             else return aggregate timeseries
         """
-        # list of variables require default components (no manual list)
-        if islistable(variable) and components is not None:
-            raise ValueError('aggregating by list of variables cannot use '
-                             'custom components')
-
-        mapping = {}
-        msg = 'cannot aggregate variable `{}` because it has no components'
-        # if single variable
-        if isstr(variable):
-            # default components to all variables one level below `variable`
-            components = components or self._variable_components(variable)
-
-            if not len(components):
-                logger.info(msg.format(variable))
-                return
-
-            for c in components:
-                mapping[c] = variable
-
-        # else, use all variables one level below `variable` as components
-        else:
-            for v in variable if islistable(variable) else [variable]:
-                _components = self._variable_components(v)
-                if not len(_components):
-                    logger.info(msg.format(v))
-
-                for c in _components:
-                    mapping[c] = v
+        _df = _aggregate(self, variable, components=components, method=method)
 
-        # rename all components to `variable` and aggregate
-        _df = self.data[self._apply_filters(variable=mapping.keys())].copy()
-        _df['variable'].replace(mapping, inplace=True)
-        _data = _aggregate(_df, [], method)
+        # return None if there is nothing to aggregate
+        if _df is None:
+            return None
 
-        # append to `self` or return as pd.Series
+        # else, append to `self` or return as `IamDataFrame`
         if append is True:
-            self.append(_data, inplace=True)
+            self.append(_df, inplace=True)
         else:
-            return _data
+            return IamDataFrame(_df)
 
     def check_aggregate(self, variable, components=None, method='sum',
                         exclude_on_fail=False, multiplier=1, **kwargs):
@@ -842,29 +834,30 @@ class IamDataFrame(object):
         kwargs: passed to `np.isclose()`
         """
         # compute aggregate from components, return None if no components
-        df_components = self.aggregate(variable, components)
+        df_components = _aggregate(self, variable, components, method)
         if df_components is None:
             return
 
         # filter and groupby data, use `pd.Series.align` for matching index
         rows = self._apply_filters(variable=variable)
-        df_variable, df_components = (
-            _aggregate(self.data[rows], [], method)
+        df_var, df_components = (
+            _group_and_agg(self.data[rows], [], method)
             .align(df_components)
         )
 
         # use `np.isclose` for checking match
-        diff = df_variable[~np.isclose(df_variable, multiplier * df_components,
-                                       **kwargs)]
+        rows = ~np.isclose(df_var, multiplier * df_components, **kwargs)
 
-        if len(diff):
+        # if aggregate and components don't match, return inconsistent data
+        if sum(rows):
             msg = '`{}` - {} of {} rows are not aggregates of components'
-            logger.info(msg.format(variable, len(diff), len(df_variable)))
+            logger.info(msg.format(variable, sum(rows), len(df_var)))
 
             if exclude_on_fail:
-                self._exclude_on_fail(diff.index.droplevel([2, 3, 4]))
+                self._exclude_on_fail(_meta_idx(df_var[rows].reset_index()))
 
-            return IamDataFrame(diff).timeseries()
+            return pd.concat([df_var[rows], df_components[rows]], axis=1,
+                             keys=(['variable', 'components']))
 
     def aggregate_region(self, variable, region='World', subregions=None,
                          components=False, method='sum', weight=None,
@@ -896,60 +889,20 @@ class IamDataFrame(object):
             append the aggregate timeseries to `self` and return None,
             else return aggregate timeseries
         """
-        if not isstr(variable) and components is not False:
-            msg = 'aggregating by list of variables with components ' \
-                  'is not supported'
-            raise ValueError(msg)
-
-        if weight is not None and components is not False:
-            msg = 'using weights and components in one operation not supported'
-            raise ValueError(msg)
-
-        # default subregions to all regions other than `region`
-        subregions = subregions or self._all_other_regions(region, variable)
-
-        if not len(subregions):
-            msg = 'cannot aggregate variable `{}` to `{}` because it does not'\
-                  ' exist in any subregion'
-            logger.info(msg.format(variable, region))
-
-            return
+        _df = _aggregate_region(
+            self, variable, region=region, subregions=subregions,
+            components=components, method=method, weight=weight
+        )
 
-        # compute aggregate over all subregions
-        subregion_df = self.filter(region=subregions)
-        rows = subregion_df._apply_filters(variable=variable)
-        if weight is None:
-            col = 'region'
-            _data = _aggregate(subregion_df.data[rows], col, method=method)
-        else:
-            weight_rows = subregion_df._apply_filters(variable=weight)
-            _data = _aggregate_weight(subregion_df.data[rows],
-                                      subregion_df.data[weight_rows], method)
-
-        # if not `components=False`, add components at the `region` level
-        if components is not False:
-            with adjust_log_level(logger):
-                region_df = self.filter(region=region)
-
-            # if `True`, auto-detect `components` at the `region` level,
-            # defaults to variables below `variable` only present in `region`
-            if components is True:
-                level = dict(level=None)
-                r_comps = region_df._variable_components(variable, **level)
-                sr_comps = subregion_df._variable_components(variable, **level)
-                components = set(r_comps).difference(sr_comps)
-
-            if len(components):
-                # rename all components to `variable` and aggregate
-                rows = region_df._apply_filters(variable=components)
-                _df = region_df.data[rows].copy()
-                _df['variable'] = variable
-                _data = _data.add(_aggregate(_df, 'region'), fill_value=0)
+        # return None if there is nothing to aggregate
+        if _df is None:
+            return None
 
+        # else, append to `self` or return as `IamDataFrame`
         if append is True:
-            self.append(_data, region=region, inplace=True)
+            self.append(_df, region=region, inplace=True)
         else:
-            return _data
+            return IamDataFrame(_df, region=region)
 
     def check_aggregate_region(self, variable, region='World', subregions=None,
                                components=False, method='sum', weight=None,
@@ -979,8 +932,9 @@ class IamDataFrame(object):
         kwargs: passed to `np.isclose()`
         """
         # compute aggregate from subregions, return None if no subregions
-        df_subregions = self.aggregate_region(variable, region, subregions,
-                                              components, method, weight)
+        df_subregions = _aggregate_region(self, variable, region, subregions,
+                                          components, method, weight)
+
         if df_subregions is None:
             return
 
@@ -992,23 +946,27 @@ class IamDataFrame(object):
             return
 
         df_region, df_subregions = (
-            _aggregate(self.data[rows], 'region')
+            _group_and_agg(self.data[rows], 'region')
             .align(df_subregions)
         )
 
         # use `np.isclose` for checking match
-        diff = df_region[~np.isclose(df_region, df_subregions, **kwargs)]
+        rows = ~np.isclose(df_region, df_subregions, **kwargs)
 
-        if len(diff):
-            msg = (
-                '`{}` - {} of {} rows are not aggregates of subregions'
-            )
-            logger.info(msg.format(variable, len(diff), len(df_region)))
+        # if region and subregions don't match, return inconsistent data
+        if sum(rows):
+            msg = '`{}` - {} of {} rows are not aggregates of subregions'
+            logger.info(msg.format(variable, sum(rows), len(df_region)))
 
             if exclude_on_fail:
-                self._exclude_on_fail(diff.index.droplevel([2, 3]))
+                self._exclude_on_fail(_meta_idx(df_region[rows].reset_index()))
 
-            return IamDataFrame(diff, region=region).timeseries()
+            _df = pd.concat(
+                [pd.concat([df_region[rows], df_subregions[rows]], axis=1,
+                           keys=(['region', 'subregions']))],
+                keys=[region], names=['region'])
+            _df.index = _df.index.reorder_levels(self._LONG_IDX)
+            return _df
 
     def downscale_region(self, variable, proxy, region='World',
                          subregions=None, append=False):
@@ -1033,12 +991,13 @@ class IamDataFrame(object):
 
         # filter relevant data, transform to `pd.Series` with appropriate index
         _df = self.data[self._apply_filters(variable=proxy, region=subregions)]
-        _proxy = _df.set_index(self._get_cols(['region', 'year'])).value
-        _total = _df.groupby(self._get_cols(['year'])).value.sum()
+        _proxy = _df.set_index(self._get_cols(['region', self.time_col])).value
+        _total = _df.groupby(self._get_cols([self.time_col])).value.sum()
 
         _value = (
             self.data[self._apply_filters(variable=variable, region=region)]
-            .set_index(self._get_cols(['variable', 'unit', 'year'])).value
+            .set_index(self._get_cols(['variable', 'unit', self.time_col]))
+            .value
         )
 
         # compute downscaled data
@@ -1071,38 +1030,45 @@ class IamDataFrame(object):
         """Return a list of columns of `self.data`"""
         return META_IDX + cols + self.extra_cols
 
-    def check_internal_consistency(self, **kwargs):
+    def check_internal_consistency(self, components=False, **kwargs):
         """Check whether a scenario ensemble is internally consistent
 
         We check that all variables are equal to the sum of their sectoral
         components and that all the regions add up to the World total. If
-        the check is passed, None is returned, otherwise a dictionary of
+        the check is passed, None is returned, otherwise a DataFrame of
         inconsistent variables is returned.
 
         Note: at the moment, this method's regional checking is limited to
         checking that all the regions sum to the World region. We cannot
-        make this more automatic unless we start to store how the regions
-        relate, see
-        [this issue](https://github.com/IAMconsortium/pyam/issues/106).
+        make this more automatic unless we store how the regions relate,
+        see [this issue](https://github.com/IAMconsortium/pyam/issues/106).
 
         Parameters
         ----------
         kwargs: passed to `np.isclose()`
+        components: bool, default False
+            passed to `check_aggregate_region)`: if `True`, use all
+            sub-categories of each `variable` included in `World` but not in
+            any of the subregions; if `False`, only aggregate variables over
+            subregions
         """
-        inconsistent_vars = {}
+        lst = []
         for variable in self.variables():
             diff_agg = self.check_aggregate(variable, **kwargs)
             if diff_agg is not None:
-                inconsistent_vars[variable + "-aggregate"] = diff_agg
+                lst.append(diff_agg)
 
             diff_regional = (
-                self.check_aggregate_region(variable, components=True,
+                self.check_aggregate_region(variable, components=components,
                                             **kwargs)
             )
             if diff_regional is not None:
-                inconsistent_vars[variable + "-regional"] = diff_regional
+                lst.append(diff_regional)
 
-        return inconsistent_vars if inconsistent_vars else None
+        if len(lst):
+            _df = pd.concat(lst, sort=True).sort_index()
+            return _df[[c for c in ['variable', 'components', 'region',
+                                    'subregions'] if c in _df.columns]]
 
     def _exclude_on_fail(self, df):
         """Assign a selection of scenarios as `exclude: True` in meta"""
@@ -1558,50 +1524,10 @@ class IamDataFrame(object):
 
 
 def _meta_idx(data):
+    """Return the `META_IDX` from `data` by index or columns"""
     return data[META_IDX].drop_duplicates().set_index(META_IDX).index
 
 
-def _aggregate(df, by, method=np.sum):
-    """Aggregate `df` by specified column(s), return indexed `pd.Series`"""
-    by = [by] if isstr(by) else by
-    cols = [c for c in list(df.columns) if c not in ['value'] + by]
-    # pick aggregator func (default: sum)
-    return df.groupby(cols)['value'].agg(_get_method_func(method))
-
-
-def _aggregate_weight(df, weight, method):
-    """Aggregate `df` by regions with weights, return indexed `pd.Series`"""
-    # only summation allowed with weights
-    if method not in ['sum', np.sum]:
-        raise ValueError('only method `np.sum` allowed for weighted average')
-
-    _data = _get_value_col(df, YEAR_IDX)
-    _weight = _get_value_col(weight, YEAR_IDX)
-
-    if not _data.index.equals(_weight.index):
-        raise ValueError('inconsistent index between variable and weight')
-
-    cols = META_IDX + ['year']
-    return (_data * _weight).groupby(cols).sum() / _weight.groupby(cols).sum()
-
-
-def _get_method_func(method):
-    """Translate a string to a known method"""
-    if not isstr(method):
-        return method
-
-    if method in KNOWN_FUNCS:
-        return KNOWN_FUNCS[method]
-
-    # raise error if `method` is a string but not in dict of known methods
-    raise ValueError('method `{}` is not a known aggregator'.format(method))
-
-
-def _get_value_col(df, cols):
-    """Return the value column as `pd.Series sorted by index"""
-    return df.set_index(cols)['value'].sort_index()
-
-
 def _raise_filter_error(col):
     raise ValueError('filter by `{}` not supported'.format(col))
 


Test patch:
diff --git a/pyam/testing.py b/pyam/testing.py
new file mode 100644
index 0000000..9a3699d
--- /dev/null
+++ b/pyam/testing.py
@@ -0,0 +1,11 @@
+from . import compare
+import pandas.testing as pdt
+
+
+def assert_frame_equal(a, b, **assert_kwargs):
+    diff = compare(a, b)
+    if not diff.empty:
+        msg = 'IamDataFrame.data are different: \n {}'
+        raise AssertionError(msg.format(diff.head()))
+
+    pdt.assert_frame_equal(a.meta, b.meta, **assert_kwargs)
diff --git a/tests/conftest.py b/tests/conftest.py
index 9aba01a..560ea15 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -68,128 +68,6 @@ REG_DF = pd.DataFrame([
 )
 
 
-mg_ascen = ['MSG-GLB', 'a_scen']
-mg_ascen_2 = ['MSG-GLB', 'a_scen_2']
-CHECK_AGG_DF = pd.DataFrame([
-    ['IMG', 'a_scen', 'R5ASIA', 'Primary Energy', 'EJ/y', 1, 6],
-    ['IMG', 'a_scen', 'R5ASIA', 'Primary Energy|Coal', 'EJ/y', 0.75, 5],
-    ['IMG', 'a_scen', 'R5ASIA', 'Primary Energy|Gas', 'EJ/y', 0.25, 1],
-    ['IMG', 'a_scen', 'R5ASIA', 'Emissions|CO2', 'Mt CO2/yr', 3, 8],
-    ['IMG', 'a_scen', 'R5ASIA', 'Emissions|CO2|Cars', 'Mt CO2/yr', 1, 3],
-    ['IMG', 'a_scen', 'R5ASIA', 'Emissions|CO2|Tar', 'Mt CO2/yr', 2, 5],
-    ['IMG', 'a_scen', 'R5REF', 'Primary Energy', 'EJ/y', 0.3, 0.6],
-    ['IMG', 'a_scen', 'R5REF', 'Primary Energy|Coal', 'EJ/y', 0.15, 0.4],
-    ['IMG', 'a_scen', 'R5REF', 'Primary Energy|Gas', 'EJ/y', 0.15, 0.2],
-    ['IMG', 'a_scen', 'R5REF', 'Emissions|CO2', 'Mt CO2/yr', 1, 1.4],
-    ['IMG', 'a_scen', 'R5REF', 'Emissions|CO2|Cars', 'Mt CO2/yr', 0.6, 0.8],
-    ['IMG', 'a_scen', 'R5REF', 'Emissions|CO2|Tar', 'Mt CO2/yr', 0.4, 0.6],
-    ['IMG', 'a_scen', 'World', 'Primary Energy', 'EJ/y', 1.3, 6.6],
-    ['IMG', 'a_scen', 'World', 'Primary Energy|Coal', 'EJ/y', 0.9, 5.4],
-    ['IMG', 'a_scen', 'World', 'Primary Energy|Gas', 'EJ/y', 0.4, 1.2],
-    ['IMG', 'a_scen', 'World', 'Emissions|CO2', 'Mt CO2/yr', 4, 9.4],
-    ['IMG', 'a_scen', 'World', 'Emissions|CO2|Cars', 'Mt CO2/yr', 1.6, 3.8],
-    ['IMG', 'a_scen', 'World', 'Emissions|CO2|Tar', 'Mt CO2/yr', 2.4, 5.6],
-    ['IMG', 'a_scen_2', 'R5ASIA', 'Primary Energy', 'EJ/y', 1.4, 6.4],
-    ['IMG', 'a_scen_2', 'R5ASIA', 'Primary Energy|Coal', 'EJ/y', 0.95, 5.2],
-    ['IMG', 'a_scen_2', 'R5ASIA', 'Primary Energy|Gas', 'EJ/y', 0.45, 1.2],
-    ['IMG', 'a_scen_2', 'R5ASIA', 'Emissions|CO2', 'Mt CO2/yr', 3.4, 8.4],
-    ['IMG', 'a_scen_2', 'R5ASIA', 'Emissions|CO2|Cars', 'Mt CO2/yr', 1.2, 3.2],
-    ['IMG', 'a_scen_2', 'R5ASIA', 'Emissions|CO2|Tar', 'Mt CO2/yr', 2.2, 5.2],
-    ['IMG', 'a_scen_2', 'R5REF', 'Primary Energy', 'EJ/y', 0.7, 1.0],
-    ['IMG', 'a_scen_2', 'R5REF', 'Primary Energy|Coal', 'EJ/y', 0.35, 0.6],
-    ['IMG', 'a_scen_2', 'R5REF', 'Primary Energy|Gas', 'EJ/y', 0.35, 0.4],
-    ['IMG', 'a_scen_2', 'R5REF', 'Emissions|CO2', 'Mt CO2/yr', 1.4, 1.8],
-    ['IMG', 'a_scen_2', 'R5REF', 'Emissions|CO2|Cars', 'Mt CO2/yr', 0.8, 1.0],
-    ['IMG', 'a_scen_2', 'R5REF', 'Emissions|CO2|Tar', 'Mt CO2/yr', 0.6, 0.8],
-    ['IMG', 'a_scen_2', 'World', 'Primary Energy', 'EJ/y', 2.1, 7.4],
-    ['IMG', 'a_scen_2', 'World', 'Primary Energy|Coal', 'EJ/y', 1.3, 5.8],
-    ['IMG', 'a_scen_2', 'World', 'Primary Energy|Gas', 'EJ/y', 0.8, 1.6],
-    ['IMG', 'a_scen_2', 'World', 'Emissions|CO2', 'Mt CO2/yr', 4.8, 10.2],
-    ['IMG', 'a_scen_2', 'World', 'Emissions|CO2|Cars', 'Mt CO2/yr', 2.0, 4.2],
-    ['IMG', 'a_scen_2', 'World', 'Emissions|CO2|Tar', 'Mt CO2/yr', 2.8, 6.0],
-    mg_ascen + ['R5ASIA', 'Primary Energy', 'EJ/y', 0.8, 5.8],
-    mg_ascen + ['R5ASIA', 'Primary Energy|Coal', 'EJ/y', 0.65, 4.9],
-    mg_ascen + ['R5ASIA', 'Primary Energy|Gas', 'EJ/y', 0.15, 0.9],
-    mg_ascen + ['R5ASIA', 'Emissions|CO2', 'Mt CO2/yr', 2.8, 7.8],
-    mg_ascen + ['R5ASIA', 'Emissions|CO2|Cars', 'Mt CO2/yr', 0.9, 2.9],
-    mg_ascen + ['R5ASIA', 'Emissions|CO2|Tar', 'Mt CO2/yr', 1.9, 4.9],
-    mg_ascen + ['R5REF', 'Primary Energy', 'EJ/y', 0.1, 0.4],
-    mg_ascen + ['R5REF', 'Primary Energy|Coal', 'EJ/y', 0.05, 0.3],
-    mg_ascen + ['R5REF', 'Primary Energy|Gas', 'EJ/y', 0.05, 0.1],
-    mg_ascen + ['R5REF', 'Emissions|CO2', 'Mt CO2/yr', 0.8, 1.2],
-    mg_ascen + ['R5REF', 'Emissions|CO2|Cars', 'Mt CO2/yr', 0.5, 0.7],
-    mg_ascen + ['R5REF', 'Emissions|CO2|Tar', 'Mt CO2/yr', 0.3, 0.5],
-    mg_ascen + ['World', 'Primary Energy', 'EJ/y', 0.9, 6.2],
-    mg_ascen + ['World', 'Primary Energy|Coal', 'EJ/y', 0.7, 5.2],
-    mg_ascen + ['World', 'Primary Energy|Gas', 'EJ/y', 0.2, 1.0],
-    mg_ascen + ['World', 'Emissions|CO2', 'Mt CO2/yr', 3.6, 9.0],
-    mg_ascen + ['World', 'Emissions|CO2|Cars', 'Mt CO2/yr', 1.4, 3.6],
-    mg_ascen + ['World', 'Emissions|CO2|Tar', 'Mt CO2/yr', 2.2, 5.4],
-    mg_ascen_2 + ['R5ASIA', 'Primary Energy', 'EJ/y', -1.4, -6.4],
-    mg_ascen_2 + ['R5ASIA', 'Primary Energy|Coal', 'EJ/y', -0.95, -5.2],
-    mg_ascen_2 + ['R5ASIA', 'Primary Energy|Gas', 'EJ/y', -0.45, -1.2],
-    mg_ascen_2 + ['R5ASIA', 'Emissions|CO2', 'Mt CO2/yr', -3.4, -8.4],
-    mg_ascen_2 + ['R5ASIA', 'Emissions|CO2|Cars', 'Mt CO2/yr', -1.2, -3.2],
-    mg_ascen_2 + ['R5ASIA', 'Emissions|CO2|Tar', 'Mt CO2/yr', -2.2, -5.2],
-    mg_ascen_2 + ['R5REF', 'Primary Energy', 'EJ/y', -0.7, -1.0],
-    mg_ascen_2 + ['R5REF', 'Primary Energy|Coal', 'EJ/y', -0.35, -0.6],
-    mg_ascen_2 + ['R5REF', 'Primary Energy|Gas', 'EJ/y', -0.35, -0.4],
-    mg_ascen_2 + ['R5REF', 'Emissions|CO2', 'Mt CO2/yr', -1.4, -1.8],
-    mg_ascen_2 + ['R5REF', 'Emissions|CO2|Cars', 'Mt CO2/yr', -0.8, -1.0],
-    mg_ascen_2 + ['R5REF', 'Emissions|CO2|Tar', 'Mt CO2/yr', -0.6, -0.8],
-    mg_ascen_2 + ['World', 'Primary Energy', 'EJ/y', -2.1, -7.4],
-    mg_ascen_2 + ['World', 'Primary Energy|Coal', 'EJ/y', -1.3, -5.8],
-    mg_ascen_2 + ['World', 'Primary Energy|Gas', 'EJ/y', -0.8, -1.6],
-    mg_ascen_2 + ['World', 'Emissions|CO2', 'Mt CO2/yr', -5.0, -10.6],
-    mg_ascen_2 + ['World', 'Emissions|CO2|Cars', 'Mt CO2/yr', -2.0, -4.2],
-    mg_ascen_2 + ['World', 'Emissions|CO2|Tar', 'Mt CO2/yr', -2.8, -6.0],
-    mg_ascen_2 + ['World', 'Emissions|CO2|Agg Agg', 'Mt CO2/yr', -0.2, -0.4],
-    mg_ascen_2 + ['World', 'Emissions|CF4', 'kt CF4/yr', 54, 56],
-    mg_ascen_2 + ['World', 'Emissions|C2F6', 'kt C2F6/yr', 32, 27],
-    mg_ascen_2 + ['World', 'Emissions|C2F6|Solvents', 'kt C2F6/yr', 30, 33],
-    mg_ascen_2 + ['World', 'Emissions|C2F6|Industry', 'kt C2F6/yr', 2, -6],
-    mg_ascen_2 + ['World', 'Emissions|CH4', 'Mt CH4/yr', 322, 217],
-    mg_ascen_2 + ['R5REF', 'Emissions|CH4', 'Mt CH4/yr', 30, 201],
-    mg_ascen_2 + ['R5ASIA', 'Emissions|CH4', 'Mt CH4/yr', 292, 16],
-],
-    columns=['model', 'scenario', 'region', 'variable', 'unit', 2005, 2010],
-)
-
-
-ms = ['AIM', 'cscen']
-CHECK_AGG_REGIONAL_DF = pd.DataFrame([
-    ms + ['World', 'Emissions|N2O', 'Mt N/yr', 1.9, 15.7],
-    ms + ['World', 'Emissions|N2O|AFOLU', 'Mt N/yr', 0.1, 0.1],
-    ms + ['World', 'Emissions|N2O|Ind', 'Mt N/yr', 1.8, 15.6],
-    ms + ['World', 'Emissions|N2O|Ind|Shipping', 'Mt N/yr', 1, 6],
-    ms + ['World', 'Emissions|N2O|Ind|Solvents', 'Mt N/yr', 1.6, 3.8],
-    ms + ['World', 'Emissions|N2O|Ind|Transport', 'Mt N/yr', -0.8, 5.8],
-    ms + ['RASIA', 'Emissions|N2O', 'Mt N/yr', 0, 5.9],
-    ms + ['RASIA', 'Emissions|N2O|Ind', 'Mt N/yr', 0, 5.9],
-    ms + ['RASIA', 'Emissions|N2O|Ind|Solvents', 'Mt N/yr', 0.8, 2.6],
-    ms + ['RASIA', 'Emissions|N2O|Ind|Transport', 'Mt N/yr', -0.8, 3.3],
-    ms + ['REUROPE', 'Emissions|N2O', 'Mt N/yr', 0.8, 3.7],
-    ms + ['REUROPE', 'Emissions|N2O|Ind', 'Mt N/yr', 0.8, 3.7],
-    ms + ['REUROPE', 'Emissions|N2O|Ind|Solvents', 'Mt N/yr', 0.8, 1.2],
-    ms + ['REUROPE', 'Emissions|N2O|Ind|Transport', 'Mt N/yr', 0, 2.5],
-    ms + ['China', 'Emissions|N2O', 'Mt N/yr', 0.2, 1.3],
-    ms + ['China', 'Emissions|N2O|Ind', 'Mt N/yr', 0.2, 1.3],
-    ms + ['China', 'Emissions|N2O|Ind|Transport', 'Mt N/yr', 0.2, 1.3],
-    ms + ['Japan', 'Emissions|N2O', 'Mt N/yr', -1, 2],
-    ms + ['Japan', 'Emissions|N2O|Ind', 'Mt N/yr', -1, 2],
-    ms + ['Japan', 'Emissions|N2O|Ind|Transport', 'Mt N/yr', -1, 2],
-    ms + ['Germany', 'Emissions|N2O', 'Mt N/yr', 2, 3],
-    ms + ['Germany', 'Emissions|N2O|Ind', 'Mt N/yr', 2, 3],
-    ms + ['Germany', 'Emissions|N2O|Ind|Transport', 'Mt N/yr', 2, 3],
-    ms + ['UK', 'Emissions|N2O', 'Mt N/yr', -2, -0.5],
-    ms + ['UK', 'Emissions|N2O|Ind', 'Mt N/yr', -2, -0.5],
-    ms + ['UK', 'Emissions|N2O|Ind|Transport', 'Mt N/yr', -2, -0.5],
-
-],
-    columns=['model', 'scenario', 'region', 'variable', 'unit', 2005, 2010],
-)
-
-
 TEST_STACKPLOT_DF = pd.DataFrame([
     ['World', 'Emissions|CO2|Energy|Oil', 'Mt CO2/yr', 2, 3.2, 2.0, 1.8],
     ['World', 'Emissions|CO2|Energy|Gas', 'Mt CO2/yr', 1.3, 1.6, 1.0, 0.7],
@@ -211,6 +89,8 @@ TEST_DTS = [datetime(2005, 6, 17), datetime(2010, 7, 21)]
 TEST_TIME_STR = ['2005-06-17', '2010-07-21']
 TEST_TIME_STR_HR = ['2005-06-17 00:00:00', '2010-07-21 12:00:00']
 
+DTS_MAPPING = {2005: TEST_DTS[0], 2010: TEST_DTS[1]}
+
 
 # minimal IamDataFrame with four different time formats
 @pytest.fixture(
@@ -243,22 +123,18 @@ def test_pd_df():
 
 
 # IamDataFrame with variable-and-region-structure for testing aggregation tools
-@pytest.fixture(scope="function")
-def aggregate_df():
-    df = IamDataFrame(model='model_a', scenario='scen_a', data=FULL_FEATURE_DF)
-    yield df
-
-
-@pytest.fixture(scope="function")
-def check_aggregate_df():
-    df = IamDataFrame(data=CHECK_AGG_DF)
-    yield df
-
-
-@pytest.fixture(scope="function")
-def check_aggregate_regional_df():
-    df = IamDataFrame(data=CHECK_AGG_REGIONAL_DF)
-    yield df
+@pytest.fixture(
+    scope="function",
+    params=[
+        None,
+        'datetime'
+    ]
+)
+def simple_df(request):
+    _df = FULL_FEATURE_DF.copy()
+    if request.param is 'datetime':
+        _df.rename(DTS_MAPPING, axis="columns", inplace=True)
+    yield IamDataFrame(model='model_a', scenario='scen_a', data=_df)
 
 
 @pytest.fixture(scope="function")
diff --git a/tests/test_core.py b/tests/test_core.py
index 5afe7ec..abf18e3 100644
--- a/tests/test_core.py
+++ b/tests/test_core.py
@@ -157,13 +157,36 @@ def test_empty_attribute(test_df_year):
     assert test_df_year.filter(model='foo').empty
 
 
+def test_equals(test_df_year):
+    test_df_year.set_meta([1, 2], name='test')
+
+    # assert that a copy (with changed index-sort) is equal
+    df = test_df_year.copy()
+    df.data = df.data.sort_values(by='value')
+    assert test_df_year.equals(df)
+
+    # assert that adding a new timeseries is not equal
+    df = test_df_year.rename(variable={'Primary Energy': 'foo'}, append=True)
+    assert not test_df_year.equals(df)
+
+    # assert that adding a new meta indicator is not equal
+    df = test_df_year.copy()
+    df.set_meta(['foo', ' bar'], name='string')
+    assert not test_df_year.equals(df)
+
+
+def test_equals_raises(test_pd_df):
+    df = IamDataFrame(test_pd_df)
+    pytest.raises(ValueError, df.equals, test_pd_df)
+
+
 def test_get_item(test_df):
     assert test_df['model'].unique() == ['model_a']
 
 
 def test_model(test_df):
-    pd.testing.assert_series_equal(test_df.models(),
-                                   pd.Series(data=['model_a'], name='model'))
+    exp = pd.Series(data=['model_a'], name='model')
+    pd.testing.assert_series_equal(test_df.models(), exp)
 
 
 def test_scenario(test_df):
diff --git a/tests/test_feature_aggregate.py b/tests/test_feature_aggregate.py
index 46214e1..ec3f85d 100644
--- a/tests/test_feature_aggregate.py
+++ b/tests/test_feature_aggregate.py
@@ -4,9 +4,9 @@ import logging
 import numpy as np
 import pandas as pd
 from pyam import check_aggregate, IamDataFrame, IAMC_IDX
+from pyam.testing import assert_frame_equal
 
-from conftest import TEST_DTS
-
+from conftest import DTS_MAPPING
 
 LONG_IDX = IAMC_IDX + ['year']
 
@@ -33,449 +33,264 @@ CO2_MAX_DF = pd.DataFrame([
     columns=LONG_IDX + ['value']
 )
 
-REG_IDX = ['model', 'scenario', 'variable', 'unit', 'year']
-
 PRICE_MAX_DF = pd.DataFrame([
-    ['model_a', 'scen_a', 'Price|Carbon', 'USD/tCO2', 2005, 10.0],
-    ['model_a', 'scen_a', 'Price|Carbon', 'USD/tCO2', 2010, 30.0],
+    ['model_a', 'scen_a', 'World', 'Price|Carbon', 'USD/tCO2', 2005, 10.0],
+    ['model_a', 'scen_a', 'World', 'Price|Carbon', 'USD/tCO2', 2010, 30.0],
 ],
-    columns=REG_IDX + ['value']
+    columns=LONG_IDX + ['value']
 )
 
 
-def test_aggregate(aggregate_df):
-    df = aggregate_df
-
-    # primary energy is a direct sum (within each region)
-    assert df.check_aggregate('Primary Energy') is None
-
-    # rename sub-category to test setting components as list
-    _df = df.rename(variable={'Primary Energy|Wind': 'foo'})
-    assert _df.check_aggregate('Primary Energy') is not None
-    components = ['Primary Energy|Coal', 'foo']
-    assert _df.check_aggregate('Primary Energy', components=components) is None
+@pytest.mark.parametrize("variable,data", (
+    ('Primary Energy', PE_MAX_DF),
+    (['Primary Energy', 'Emissions|CO2'], pd.concat([PE_MAX_DF, CO2_MAX_DF])),
+))
+def test_aggregate(simple_df, variable, data):
+    # check that `variable` is a a direct sum and matches given total
+    exp = simple_df.filter(variable=variable)
+    assert_frame_equal(simple_df.aggregate(variable), exp)
 
     # use other method (max) both as string and passing the function
-    exp = PE_MAX_DF.set_index(LONG_IDX).value
+    _df = data.copy()
+    if simple_df.time_col == 'time':
+        _df.year = _df.year.replace(DTS_MAPPING)
+        _df.rename({'year': 'time'}, axis='columns', inplace=True)
+    exp = IamDataFrame(_df)
+    for m in ['max', np.max]:
+        assert_frame_equal(simple_df.aggregate(variable, method=m), exp)
+
+
+def test_check_aggregate(simple_df):
+    # assert that `check_aggregate` returns None for full data
+    assert simple_df.check_aggregate('Primary Energy') is None
+
+    # assert that `check_aggregate` returns non-matching data
+    obs = (
+        simple_df
+        .filter(variable='Primary Energy|Coal', region='World', keep=False)
+        .check_aggregate('Primary Energy')
+    )
+    exp = pd.DataFrame([[12., 3.], [15., 5.]])
+    np.testing.assert_array_equal(obs.values, exp.values)
 
-    obs = df.aggregate('Primary Energy', method='max')
-    pd.testing.assert_series_equal(obs, exp)
 
-    obs = df.aggregate('Primary Energy', method=np.max)
-    pd.testing.assert_series_equal(obs, exp)
+def test_check_aggregate_top_level(simple_df):
+    # assert that `check_aggregate` returns None for full data
+    assert check_aggregate(simple_df, variable='Primary Energy',
+                           year=2005) is None
 
-    # using illegal method raises an error
-    pytest.raises(ValueError, df.aggregate, 'Primary Energy', method='foo')
+    # duplicate scenario, assert `check_aggregate` returns non-matching data
+    _df = (
+        simple_df
+        .rename(scenario={'scen_a': 'foo'}, append=True)
+        .filter(scenario='foo', variable='Primary Energy|Coal', keep=False)
+    )
 
+    obs = check_aggregate(_df, variable='Primary Energy', year=2005,
+                          exclude_on_fail=True)
+    exp = pd.DataFrame([[12., 3.], [8., 2.], [4., 1.]])
+    np.testing.assert_array_equal(obs.values, exp.values)
 
-def test_aggregate_by_list(aggregate_df):
-    df = aggregate_df
-    var_list = ['Primary Energy', 'Emissions|CO2']
+    # assert that scenario `foo` has correctly been assigned as `exclude=True`
+    np.testing.assert_array_equal(_df.meta.exclude.values, [True, False])
 
-    # primary energy and emissions are a direct sum (within each region)
-    assert df.check_aggregate(var_list) is None
 
-    # use other method (max) both as string and passing the function
-    exp = (
-        pd.concat([PE_MAX_DF, CO2_MAX_DF])
-        .set_index(LONG_IDX).value
-        .sort_index()
-    )
+@pytest.mark.parametrize("variable", (
+    ('Primary Energy'),
+    (['Primary Energy', 'Emissions|CO2']),
+))
+def test_aggregate_append(simple_df, variable):
+    # remove `variable`, do aggregate and append, check equality to original
+    _df = simple_df.filter(variable=variable, keep=False)
+    _df.aggregate(variable, append=True)
+    assert_frame_equal(_df, simple_df)
 
-    obs = df.aggregate(var_list, method='max')
-    pd.testing.assert_series_equal(obs, exp)
 
-    obs = df.aggregate(var_list, method=np.max)
-    pd.testing.assert_series_equal(obs, exp)
+def test_aggregate_with_components(simple_df):
+    # rename sub-category to test setting components explicitly as list
+    df = simple_df.rename(variable={'Primary Energy|Wind': 'foo'})
+    assert df.check_aggregate('Primary Energy') is not None
+    components = ['Primary Energy|Coal', 'foo']
+    assert df.check_aggregate('Primary Energy', components=components) is None
+
 
+def test_aggregate_by_list_with_components_raises(simple_df):
     # using list of variables and components raises an error
+    v = ['Primary Energy', 'Emissions|CO2']
     components = ['Primary Energy|Coal', 'Primary Energy|Wind']
-    pytest.raises(ValueError, df.aggregate, var_list, components=components)
-
-
-def test_aggregate_region(aggregate_df):
-    df = aggregate_df
+    pytest.raises(ValueError, simple_df.aggregate, v, components=components)
 
-    # primary energy is a direct sum (across regions)
-    assert df.check_aggregate_region('Primary Energy') is None
 
-    # CO2 emissions have "bunkers" only defined at the region level
-    v = 'Emissions|CO2'
-    assert df.check_aggregate_region(v) is not None
-    assert df.check_aggregate_region(v, components=True) is None
+def test_aggregate_empty(simple_df):
+    assert simple_df.aggregate('foo') is None
 
-    # rename emissions of bunker to test setting components as list
-    _df = df.rename(variable={'Emissions|CO2|Bunkers': 'foo'})
-    assert _df.check_aggregate_region(v, components=['foo']) is None
 
-    # carbon price shouldn't be summed but be weighted by emissions
-    assert df.check_aggregate_region('Price|Carbon') is not None
-    assert df.check_aggregate_region('Price|Carbon', weight=v) is None
+def test_aggregate_unknown_method(simple_df):
+    # using unknown string as method raises an error
+    v = 'Primary Energy'
+    pytest.raises(ValueError, simple_df.aggregate_region, v, method='foo')
 
-    # inconsistent index of variable and weight raises an error
-    _df = df.filter(variable='Emissions|CO2', region='reg_b', keep=False)
-    pytest.raises(ValueError, _df.aggregate_region, 'Price|Carbon',
-                  weight='Emissions|CO2')
-
-    # setting both weight and components raises an error
-    pytest.raises(ValueError, df.aggregate_region, v, components=True,
-                  weight='bar')
 
-    # use other method (max) both as string and passing the function
-    exp = PRICE_MAX_DF.set_index(REG_IDX).value
-    obs = df.aggregate_region('Price|Carbon', method='max')
-    pd.testing.assert_series_equal(obs, exp)
+@pytest.mark.parametrize("variable", (
+    ('Primary Energy'),
+    (['Primary Energy', 'Primary Energy|Coal', 'Primary Energy|Wind']),
+))
+def test_aggregate_region(simple_df, variable):
+    # check that `variable` is a a direct sum across regions
+    exp = simple_df.filter(variable=variable, region='World')
+    assert_frame_equal(simple_df.aggregate_region(variable), exp)
+
+    # check custom `region` (will include `World`, so double-count values)
+    foo = exp.rename(region={'World': 'foo'})
+    foo.data.value = foo.data.value * 2
+    assert_frame_equal(simple_df.aggregate_region(variable, region='foo'), foo)
+
+
+def test_aggregate_region_log(simple_df, caplog):
+    # verify that `check_aggregate_region()` writes log on empty assertion
+    caplog.set_level(logging.INFO, logger="pyam._aggregate")
+    simple_df.aggregate_region('foo')
+    msg = ("cannot aggregate variable `foo` to `World` "
+           "because it does not exist in any subregion")
+    idx = caplog.messages.index(msg)
+    assert caplog.records[idx].levelname == "INFO"
+
+
+def test_check_aggregate_region(simple_df):
+    # assert that `check_aggregate_region` returns None for full data
+    assert simple_df.check_aggregate_region('Primary Energy') is None
+
+    # assert that `check_aggregate_region` returns non-matching data
+    obs = (
+        simple_df
+        .filter(variable='Primary Energy', region='reg_a', keep=False)
+        .check_aggregate_region('Primary Energy')
+    )
+    exp = pd.DataFrame([[12., 4.], [15., 6.]])
+    np.testing.assert_array_equal(obs.values, exp.values)
 
-    obs = df.aggregate_region('Price|Carbon', method=np.max)
-    pd.testing.assert_series_equal(obs, exp)
 
-    # using illegal method raises an error
-    pytest.raises(ValueError, df.aggregate_region, v, method='foo')
+def test_check_aggregate_region_log(simple_df, caplog):
+    # verify that `check_aggregate_region()` writes log on empty assertion
+    caplog.set_level(logging.INFO, logger="pyam.core")
+    (
+        simple_df.filter(variable='Primary Energy', region='World', keep=False)
+        .check_aggregate_region('Primary Energy')
+    )
+    print(caplog.messages)
+    msg = "variable `Primary Energy` does not exist in region `World`"
+    idx = caplog.messages.index(msg)
+    assert caplog.records[idx].levelname == "INFO"
 
-    # using weight and method other than 'sum' raises an error
-    pytest.raises(ValueError, df.aggregate_region, v, method='max',
-                  weight='bar')
 
+@pytest.mark.parametrize("variable", (
+    ('Primary Energy'),
+    (['Primary Energy', 'Primary Energy|Coal', 'Primary Energy|Wind']),
+))
+def test_aggregate_region_append(simple_df, variable):
+    # remove `variable`, do aggregate and append, check equality to original
+    _df = simple_df.filter(variable=variable, region='World', keep=False)
+    _df.aggregate_region(variable, append=True)
+    assert_frame_equal(_df, simple_df)
 
-def test_aggregate_region_by_list(aggregate_df):
-    df = aggregate_df
-    var_list = ['Primary Energy', 'Primary Energy|Coal', 'Primary Energy|Wind']
 
-    # primary energy and sub-categories are a direct sum (across regions)
-    assert df.check_aggregate_region(var_list) is None
+@pytest.mark.parametrize("variable", (
+    ('Primary Energy'),
+    (['Primary Energy', 'Primary Energy|Coal', 'Primary Energy|Wind']),
+))
+def test_aggregate_region_with_subregions(simple_df, variable):
+    # check that custom `subregions` works (assumes only `reg_a` is in `World`)
+    exp = (
+        simple_df.filter(variable=variable, region='reg_a')
+        .rename(region={'reg_a': 'World'})
+    )
+    obs = simple_df.aggregate_region(variable, subregions='reg_a')
+    assert_frame_equal(obs, exp)
 
-    # emissions and carbon price are _not_ a direct sum (across regions)
-    var_list = ['Price|Carbon', 'Emissions|CO2']
-    assert df.check_aggregate_region(var_list) is not None
+    # check that both custom `region` and `subregions` work
+    foo = exp.rename(region={'World': 'foo'})
+    obs = simple_df.aggregate_region(variable, region='foo',
+                                     subregions='reg_a')
+    assert_frame_equal(obs, foo)
 
-    # using list of variables and components raises an error
-    pytest.raises(ValueError, df.aggregate_region, var_list, components=True)
+    # check that invalid list of subregions returns empty
+    assert simple_df.aggregate_region(variable, subregions=['reg_c']).empty
 
-    # using list of variables and weight raises an error (inconsistent weight)
-    pytest.raises(ValueError, df.aggregate_region, var_list, weight=True)
 
+@pytest.mark.parametrize("variable,data", (
+    ('Price|Carbon', PRICE_MAX_DF),
+    (['Price|Carbon', 'Emissions|CO2'], pd.concat([PRICE_MAX_DF, CO2_MAX_DF]))
+))
+def test_aggregate_region_with_other_method(simple_df, variable, data):
     # use other method (max) both as string and passing the function
-    _co2_df = CO2_MAX_DF[CO2_MAX_DF.region == 'World'].drop(columns='region')
-    exp = pd.concat([_co2_df, PRICE_MAX_DF]).set_index(REG_IDX).value
-
-    obs = df.aggregate_region(var_list, method='max')
-    pd.testing.assert_series_equal(obs, exp)
-
-    obs = df.aggregate_region(var_list, method=np.max)
-    pd.testing.assert_series_equal(obs, exp)
-
-
-def test_missing_region(check_aggregate_df):
-    # for now, this test makes sure that this operation works as expected
-    exp = check_aggregate_df.aggregate_region('Primary Energy', region='foo')
-    assert len(exp) == 8
-    # # this test should be updated to the below after the return type of
-    # # aggregate_region() is updated
-    # exp = check_aggregate_df.aggregate_region(
-    #     'Primary Energy', region='foo', append=False
-    # ).data
-    # check_aggregate_df.aggregate_region(
-    #     'Primary Energy', region='foo', append=True
-    # )
-    # obs = check_aggregate_df.filter(region='foo').data
-    # assert len(exp) > 0
-    # pd.testing.assert_frame_equal(obs.reset_index(drop=True),
-    #                               exp.reset_index(drop=True))
-
-
-def test_aggregate_region_extra_subregion():
-    cols = ['model', 'scenario', 'region', 'variable', 'unit', 2005, 2010]
-    data = pd.DataFrame([
-        ['model_a', 'scen_a', 'foo', 'Primary Energy', 'EJ/y', 1, 6],
-        ['model_a', 'scen_a', 'bar', 'Primary Energy', 'EJ/y', 0.75, 5]],
-        columns=cols)
-    df = IamDataFrame(data=data)
-    obs = df.aggregate_region(variable='Primary Energy',
-                              region='R5ASIA',
-                              subregions=['foo', 'bar', 'baz'],
-                              components=[], append=False)
-    assert len(obs) == 2
-
-
-def test_aggregate_region_missing_all_subregions():
-    cols = ['model', 'scenario', 'region', 'variable', 'unit', 2005, 2010]
-    data = pd.DataFrame([
-        ['model_a', 'scen_a', 'foo', 'Primary Energy', 'EJ/y', 1, 6],
-        ['model_a', 'scen_a', 'bar', 'Primary Energy', 'EJ/y', 0.75, 5]],
-        columns=cols)
-    df = IamDataFrame(data=data)
-    obs = df.aggregate_region(variable='Primary Energy',
-                              region='R5ASIA',
-                              subregions=['China', 'Vietnam', 'Japan']
-                              )
-    assert len(obs) == 0
-
-
-def test_do_aggregate_append(test_df):
-    test_df.rename({'variable': {'Primary Energy': 'Primary Energy|Gas'}},
-                   inplace=True)
-    test_df.aggregate('Primary Energy', append=True)
-    df = test_df.filter(variable='Primary Energy')
-
-    times = [2005, 2010] if "year" in test_df.data else TEST_DTS
-    exp = pd.DataFrame([
-        ['model_a', 'scen_a', 'World', 'Primary Energy', 'EJ/y', 1.5, 9.],
-        ['model_a', 'scen_b', 'World', 'Primary Energy', 'EJ/y', 2, 7],
-    ],
-        columns=['model', 'scenario', 'region', 'variable', 'unit'] + times
-    ).set_index(IAMC_IDX)
-    if "year" in test_df.data:
-        exp.columns = list(map(int, exp.columns))
-    else:
-        df.data.time = df.data.time.dt.normalize()
-        exp.columns = pd.to_datetime(exp.columns)
-
-    pd.testing.assert_frame_equal(df.timeseries(), exp)
-
-
-def test_aggregate_unknown_method(reg_df):
-    pytest.raises(ValueError, reg_df.aggregate_region, 'Primary Energy',
-                  method='foo')
-
-
-def test_check_aggregate_pass(check_aggregate_df):
-    obs = check_aggregate_df.filter(
-        scenario='a_scen'
-    ).check_aggregate('Primary Energy')
-    assert obs is None
-
-
-def test_check_internal_consistency_no_world_for_variable(
-    check_aggregate_df, caplog
-):
-    assert check_aggregate_df.check_internal_consistency() is None
-    test_df = check_aggregate_df.filter(
-        variable='Emissions|CH4', region='World', keep=False
-    )
-    caplog.set_level(logging.INFO, logger="pyam.core")
-    test_df.check_internal_consistency()
-    warn_idx = caplog.messages.index("variable `Emissions|CH4` does not exist "
-                                     "in region `World`")
-    assert caplog.records[warn_idx].levelname == "INFO"
+    _df = data.copy()
+    if simple_df.time_col == 'time':
+        _df.year = _df.year.replace(DTS_MAPPING)
+        _df.rename({'year': 'time'}, axis='columns', inplace=True)
+    exp = IamDataFrame(_df).filter(region='World')
+    for m in ['max', np.max]:
+        assert_frame_equal(simple_df.aggregate_region(variable, method=m), exp)
 
 
-def test_check_aggregate_fail(test_df):
-    obs = test_df.check_aggregate('Primary Energy', exclude_on_fail=True)
-    assert len(obs.columns) == 2
-    assert obs.index.get_values()[0] == (
-        'model_a', 'scen_a', 'World', 'Primary Energy', 'EJ/y'
-    )
-
+def test_aggregate_region_with_components(simple_df):
+    # CO2 emissions have "bunkers" only defined at the region level
+    v = 'Emissions|CO2'
+    assert simple_df.check_aggregate_region(v) is not None
+    assert simple_df.check_aggregate_region(v, components=True) is None
 
-def test_check_aggregate_top_level(test_df):
-    obs = check_aggregate(test_df, variable='Primary Energy', year=2005)
-    assert len(obs.columns) == 1
-    assert obs.index.get_values()[0] == (
-        'model_a', 'scen_a', 'World', 'Primary Energy', 'EJ/y'
-    )
+    # rename emissions of bunker to test setting components as list
+    _df = simple_df.rename(variable={'Emissions|CO2|Bunkers': 'foo'})
+    assert _df.check_aggregate_region(v, components=['foo']) is None
 
 
-def test_df_check_aggregate_pass(check_aggregate_df):
-    obs = check_aggregate_df.check_aggregate('Primary Energy')
-    assert obs is None
-
-    for variable in check_aggregate_df.variables():
-        obs = check_aggregate_df.check_aggregate(variable)
-        assert obs is None
-
-
-def test_df_check_aggregate_region_pass(check_aggregate_df):
-    comp = dict(components=True)
-    obs = check_aggregate_df.check_aggregate_region('Primary Energy', **comp)
-    assert obs is None
-
-    for variable in check_aggregate_df.variables():
-        obs = check_aggregate_df.check_aggregate_region(variable, **comp)
-        assert obs is None
-
-
-def run_check_agg_fail(pyam_df, tweak_dict, test_type):
-    mr = pyam_df.data.model == tweak_dict['model']
-    sr = pyam_df.data.scenario == tweak_dict['scenario']
-    rr = pyam_df.data.region == tweak_dict['region']
-    vr = pyam_df.data.variable == tweak_dict['variable']
-    ur = pyam_df.data.unit == tweak_dict['unit']
-
-    row_to_tweak = mr & sr & rr & vr & ur
-    assert row_to_tweak.any()
-
-    pyam_df.data.value.iloc[np.where(row_to_tweak)[0]] *= 0.99
-
-    # the error variable is always the top level one
-    expected_index = tweak_dict
-    agg_test = test_type == 'aggregate'
-    region_world_only_contrib = test_type == 'region-world-only-contrib'
-    if agg_test or region_world_only_contrib:
-        expected_index['variable'] = '|'.join(
-            expected_index['variable'].split('|')[:2]
-        )
-    elif 'region' in test_type:
-        expected_index['region'] = 'World'
-
-    expected_index = [v for k, v in expected_index.items()]
-
-    for variable in pyam_df.variables():
-        if test_type == 'aggregate':
-            obs = pyam_df.check_aggregate(
-                variable,
-            )
-        elif 'region' in test_type:
-            obs = pyam_df.check_aggregate_region(
-                variable, components=True
-            )
-
-        if obs is not None:
-            assert len(obs.columns) == 2
-            assert set(obs.index.get_values()[0]) == set(expected_index)
-
-
-def test_df_check_aggregate_fail(check_aggregate_df):
-    to_tweak = {
-        'model': 'IMG',
-        'scenario': 'a_scen_2',
-        'region': 'R5REF',
-        'variable': 'Emissions|CO2',
-        'unit': 'Mt CO2/yr',
-    }
-    run_check_agg_fail(check_aggregate_df, to_tweak, 'aggregate')
-
-
-def test_df_check_aggregate_fail_no_regions(check_aggregate_df):
-    to_tweak = {
-        'model': 'MSG-GLB',
-        'scenario': 'a_scen_2',
-        'region': 'World',
-        'variable': 'Emissions|C2F6|Solvents',
-        'unit': 'kt C2F6/yr',
-    }
-    run_check_agg_fail(check_aggregate_df, to_tweak, 'aggregate')
-
-
-def test_df_check_aggregate_region_fail(check_aggregate_df):
-    to_tweak = {
-        'model': 'IMG',
-        'scenario': 'a_scen_2',
-        'region': 'World',
-        'variable': 'Emissions|CO2',
-        'unit': 'Mt CO2/yr',
-    }
-    run_check_agg_fail(check_aggregate_df, to_tweak, 'region')
-
-
-def test_df_check_aggregate_region_fail_no_subsector(check_aggregate_df):
-    to_tweak = {
-        'model': 'MSG-GLB',
-        'scenario': 'a_scen_2',
-        'region': 'R5REF',
-        'variable': 'Emissions|CH4',
-        'unit': 'Mt CH4/yr',
-    }
-    run_check_agg_fail(check_aggregate_df, to_tweak, 'region')
-
-
-def test_df_check_aggregate_region_fail_world_only_var(check_aggregate_df):
-    to_tweak = {
-        'model': 'MSG-GLB',
-        'scenario': 'a_scen_2',
-        'region': 'World',
-        'variable': 'Emissions|CO2|Agg Agg',
-        'unit': 'Mt CO2/yr',
-    }
-
-    run_check_agg_fail(
-        check_aggregate_df, to_tweak, 'region-world-only-contrib'
-    )
+def test_aggregate_region_with_weights(simple_df):
+    # carbon price shouldn't be summed but be weighted by emissions
+    v = 'Price|Carbon'
+    w = 'Emissions|CO2'
+    assert simple_df.check_aggregate_region(v) is not None
+    assert simple_df.check_aggregate_region(v, weight=w) is None
 
+    exp = simple_df.filter(variable=v, region='World')
+    assert_frame_equal(simple_df.aggregate_region(v, weight=w), exp)
 
-def test_df_check_aggregate_region_errors(check_aggregate_regional_df):
-    # these tests should fail because our dataframe has continents and regions
-    # so checking without providing components leads to double counting and
-    # hence failure
-    obs = check_aggregate_regional_df.check_aggregate_region(
-        'Emissions|N2O', 'World'
-    )
+    # inconsistent index of variable and weight raises an error
+    _df = simple_df.filter(variable=w, region='reg_b', keep=False)
+    pytest.raises(ValueError, _df.aggregate_region, v, weight=w)
 
-    assert len(obs.columns) == 2
-    assert obs.index.get_values()[0] == (
-        'AIM', 'cscen', 'World', 'Emissions|N2O', 'Mt N/yr'
-    )
+    # using weight and method other than 'sum' raises an error
+    pytest.raises(ValueError, simple_df.aggregate_region, v, method='max',
+                  weight='bar')
 
-    obs = check_aggregate_regional_df.check_aggregate_region(
-        'Emissions|N2O', 'REUROPE'
-    )
 
-    assert len(obs.columns) == 2
-    assert obs.index.get_values()[0] == (
-        'AIM', 'cscen', 'REUROPE', 'Emissions|N2O', 'Mt N/yr'
-    )
+def test_aggregate_region_with_components_and_weights_raises(simple_df):
+    # setting both weight and components raises an error
+    pytest.raises(ValueError, simple_df.aggregate_region, 'Emissions|CO2',
+                  components=True, weight='bar')
 
 
-def test_df_check_aggregate_region_components(check_aggregate_regional_df):
-    obs = check_aggregate_regional_df.check_aggregate_region(
-        'Emissions|N2O', 'World', subregions=['REUROPE', 'RASIA'],
-        components=True
-    )
-    assert obs is None
+def test_aggregate_region_empty(simple_df):
+    assert simple_df.aggregate_region('foo') is None
 
-    obs = check_aggregate_regional_df.check_aggregate_region(
-        'Emissions|N2O|Ind|Solvents', 'World', subregions=['REUROPE', 'RASIA'],
-        components=True
-    )
-    assert obs is None
 
-    obs = check_aggregate_regional_df.check_aggregate_region(
-        'Emissions|N2O', 'REUROPE', subregions=['Germany', 'UK'],
-        components=True
-    )
-    assert obs is None
+def test_aggregate_region_unknown_method(simple_df):
+    # using unknown string as method raises an error
+    v = 'Emissions|CO2'
+    pytest.raises(ValueError, simple_df.aggregate_region, v, method='foo')
 
-    obs = check_aggregate_regional_df.check_aggregate_region(
-        'Emissions|N2O', 'RASIA', subregions=['China', 'Japan'],
-        components=True
-    )
-    assert obs is None
 
-    obs = check_aggregate_regional_df.check_aggregate_region(
-        'Emissions|N2O|Ind|Transport', 'REUROPE', subregions=['Germany', 'UK'],
-        components=True
-    )
-    assert obs is None
+def test_check_internal_consistency(simple_df):
+    _df = simple_df.filter(variable='Price|Carbon', keep=False)
 
+    # assert that test data is consistent (except for `Price|Carbon`)
+    assert _df.check_internal_consistency(components=True) is None
 
-@pytest.mark.parametrize("components,exp_vals", (
-    # should find sub-components including nested bunkers
-    (True, [1.9, 15.7]),
-    # should only add AFOLU onto regional sum, not Shipping emissions
-    (["Emissions|N2O|AFOLU"], [0.9, 9.7]),
-    # specifying Ind leads to double counting (and not skipping AFOLU) but as
-    # it's user specified no warning etc. is raised
-    (["Emissions|N2O|Ind"], [2.6, 25.2]),
-))
-def test_aggregate_region_components_handling(check_aggregate_regional_df,
-                                              components, exp_vals):
-    tdf = check_aggregate_regional_df.filter(variable="*N2O*")
-    # only get Europe and Asia to avoid double counting
-    res = tdf.aggregate_region("Emissions|N2O", components=components,
-                               subregions=["REUROPE", "RASIA"])
-    exp_idx = pd.MultiIndex.from_product(
-        [["AIM"], ["cscen"], ['Emissions|N2O'], ["Mt N/yr"], [2005, 2010]],
-        names=["model", "scenario", "variable", "unit", "year"]
+    # assert removing a specific subsector causes inconsistencies
+    obs = (
+        _df.filter(variable='Primary Energy|Coal', region='reg_a', keep=False)
+        .check_internal_consistency(components=True)
     )
-    exp = pd.Series(exp_vals, index=exp_idx)
-    exp.name = "value"
-
-    pd.testing.assert_series_equal(res, exp)
 
-
-def test_check_aggregate_region_no_world(check_aggregate_regional_df, caplog):
-    test_df = check_aggregate_regional_df.filter(region='World', keep=False)
-    caplog.set_level(logging.INFO, logger="pyam.core")
-    test_df.check_aggregate_region('Emissions|N2O', region='World')
-    warn_idx = caplog.messages.index("variable `Emissions|N2O` does not exist "
-                                     "in region `World`")
-    assert caplog.records[warn_idx].levelname == "INFO"
+    # test reported inconsistency
+    exp = pd.DataFrame([[np.nan, np.nan, 9., 3.], [np.nan, np.nan, 10., 4.],
+                        [8., 2., np.nan, np.nan], [9., 3., np.nan, np.nan]])
+    np.testing.assert_array_equal(obs.values, exp.values)
diff --git a/tests/test_feature_downscale.py b/tests/test_feature_downscale.py
index 1c1f599..9509830 100644
--- a/tests/test_feature_downscale.py
+++ b/tests/test_feature_downscale.py
@@ -1,26 +1,21 @@
 import pytest
-import pandas as pd
-import pyam
+from pyam.testing import assert_frame_equal
 
 
 @pytest.mark.parametrize("variable", (
     ('Primary Energy'),
     (['Primary Energy', 'Primary Energy|Coal']),
 ))
-def test_downscale_region(aggregate_df, variable):
-    df = aggregate_df
-    df.set_meta([1], name='test')
-
+def test_downscale_region(simple_df, variable):
+    simple_df.set_meta([1], name='test')
     regions = ['reg_a', 'reg_b']
 
     # return as new IamDataFrame
-    obs_df = df.downscale_region(variable, proxy='Population')
-    exp_df = df.filter(variable=variable, region=regions)
-    assert pyam.compare(obs_df, exp_df).empty
-    pd.testing.assert_frame_equal(obs_df.meta, exp_df.meta)
+    obs = simple_df.downscale_region(variable, proxy='Population')
+    exp = simple_df.filter(variable=variable, region=regions)
+    assert_frame_equal(exp, obs)
 
     # append to `self` (after removing to-be-downscaled timeseries)
-    inplace_df = df.filter(variable=variable, region=regions, keep=False)
-    inplace_df.downscale_region(variable, proxy='Population', append=True)
-    assert pyam.compare(inplace_df, df).empty
-    pd.testing.assert_frame_equal(inplace_df.meta, df.meta)
+    inplace = simple_df.filter(variable=variable, region=regions, keep=False)
+    inplace.downscale_region(variable, proxy='Population', append=True)
+    assert_frame_equal(inplace, simple_df)
diff --git a/tests/test_tutorials.py b/tests/test_tutorials.py
index a6af555..1c731c7 100644
--- a/tests/test_tutorials.py
+++ b/tests/test_tutorials.py
@@ -24,7 +24,7 @@ pandoc_reason = 'requires Pandoc to be installed'
 
 tut_path = os.path.join(here, '..', 'doc', 'source', 'tutorials')
 
-# taken from the execellent example here:
+# taken from the excellent example here:
 # https://blog.thedataincubator.com/2016/06/testing-jupyter-notebooks/
 
 
@@ -68,8 +68,8 @@ def test_pyam_first_steps(capsys):
 
 @pytest.mark.skipif(not jupyter_installed, reason=jupyter_reason)
 @pytest.mark.skipif(not pandoc_installed, reason=pandoc_reason)
-def test_checking_consistency():
-    fname = os.path.join(tut_path, 'checking_consistency.ipynb')
+def test_aggregating_downscaling_consistency():
+    fname = os.path.join(tut_path, 'aggregating_downscaling_consistency.ipynb')
     nb, errors = _notebook_run(fname)
     assert errors == []