dataesr/scientific-paragraphs-categorization huggingface dataset:
ESPON EGTC is a regional planning agency for the European Commission. In European larger cities decent and affordable housing is increasingly hard to get access to: the goal of the study is to inform the increased and unequal affordability gap at the local geographical level, with tools for comparison between cities, and within cities. The aim of the original ESPON report published in 2019 was to inform European public policies in territorial cohesion. The report has been drafted with a description and discussion of the results, in an analysis of the unequal spatial patterns of unaffordability in Europe. Information regarding the reproducibility of the work and data collection has also been added to the report, as appendix. The aim of this data paper is to complement the report with a full description of the database constructed, its metadata, the possible usages for further research, and procedures for reproducibility and future additions to the database.
We do so and present the structure of the database, how it has been constructed in a reproducible manner ; we document the validation process, the strengths and limitations of the data provided, and document the reproducibility of the workflow. The database stored on the Nakala open access respository comes with a fully reproducible example documented as a RMarkdown document 1. The database has been structured so as to answer several issues altogether, that are the theoretical backbones of the research. One dimension of the problem is unequal access to housing opportunity with regard to income and wealth of households. Another dimension is that this gap has widened during the last decades. Since the 1990s, housing prices have on average increased faster than the income of residents and buyers in major post-industrial city-regions, but this trend is not ubiquitous and there are some exceptions.
The methodology has been applied to 10 case-studies Functional Urban Areas (FUAs) covering 4 countries of the ESPON Area, among which one crossborder region: Geneva (Switzerland), Annecy-Annemasse, Avignon and Paris (France), Madrid, Barcelona and Palma de Majorca (Spain) and Warsaw, Łódź and Krakow (Poland). This database does not cover the entire set of FUAs, but has been designed as a proof of concept, to envision a wider study that could bring together experts from different countries in Europe, to build a more thorough analysis. Open code and transferable methods are documented and disseminated with the ESPON study, to allow further implementation. Section 2 provides a literature review, and discusses the conceptual contribution of the database in the wider context of a policy-oriented research on housing at a European level.
Section 3 presents the methods used for the data collection: data sources, spatial coverage and case-studies, aggregation and harmonization procedures. It details the conceptual and theoretical models applied to conduct the data collection, and the methodological framework applied. We particularly insist on the effort to bridge the gap by harmonizing conventional and institutional data sources, with unconventional sources from the web ("big data"), to map affordability, along with the quality control and data validation procedures. Section 4 is dedicated to presenting a possible use of the dataset, by mapping and comparing local housing data and giving insights on the main findings from the data. The final section is dedicated to metadata.
Context, literature review and contribution of the database in housing research This section discusses the policy context in which the production of the database took place, and frames the issue of affordability and inflation, within the wider literature on housing regimes and inequality. We argue that the lack of harmonized local spatial data to map affordability in Europe is a major issue to better analyze and compare the situation at local geographical levels. The aim of the ESPON database is to present data and reproducible methods that contribute to filling this data gap.
This is not, however, the case in every European country, as demonstrated in a study by  Beresewicz (2015) , where data show a decreasing trend on the stock market in Poland after 2008. This affordability problem spreads across markets, with unequal effects  (Kemeny 2001; André and Chalaux 2018). The OECD Affordable Housing Database (AHD) characterizes the various conditions existing on European markets. Data show that in many European countries, price paid (through the average mortgage or rent burden) tops up the 25% of disposable income critical threshold, in Norway, Finland, Czech Republic, Sweden, the Netherlands, Greece, the UK, Denmark and Spain as for rents. For owner-occupied households, this is measured by means of the cost of mortgage, and it reaches the highest levels of disposable income in France, in Luxembourg, Ireland and Croatia.
Filling a data gap In such a context, the goal of the database is to detail the relationships between income, price and local market conditions in European FUAs. However, one major issue is the lack of harmonized spatial data to map and analyze affordability in Europe. There are plenty of institutional (tax, census), private (real-estate agents and websites) and national or local data (parcels, local tax rolls). These are not harmonized and easy to inter-operate. ESPON interest in the database has been policy oriented. However, data by OECD and Eurostat are disseminated respectively at the national and at the city levels, but the datasets are far from complete in terms of thematic and geographical objects available to accurately analyze housing dynamics. The Urban Audit survey (Eurostat) publishes some perception data on the conditions of housing, and annual statistic aggregates on price, type of structure and households' income, with an average degree of completeness of 51%, along with missing data in several countries in Europe, including France.
The OECD database characterizes affordability as national aggregates, and allows for comparisons to be made between countries  (André and Chalaux 2018). To compare housing situations between European cities, Eurostat provides some indicators mainly derived from national censuses under the topic "living conditions". It provides basic distinctions between houses and apartments, housing type (owneroccupied dwellings, households in social housing), as well as incomplete data on the average price for buyers and renters and average income. Eurostat also provides an overview of the distribution of housing conditions across the European urban hierarchy. Two shortcomings should be highlighted: (1) a lack of information regarding the definition and harmonization between databases; (2) the market segments and spatial coverages for which the aggregated information is relevant.
Data sources The data collection was organized using two categories of data sources (Figure  1 ). Conventional data sources are provided by traditional statistical offices through census for public, demographic use and policies. We gathered data from these providers mainly to inform households income, and we used local income by national statistical offices, and national income data at national level (Eurostat). We combine such classical indicators with unconventional data sources to describe housing market. Data were extracted from various platforms and sources and are often named "big data". We define as "unconventional" data sources that have not been designed and/or documented to prepare measurements, geographical processing and socio-economic research. They often derive from administration or commercial purposes.
We distinguish two kinds of unconventional data providers. Institutional data providers on the one hand (Chambre des Notaires in France , Wuestpartner in Switzerland). Such commercial providers distribute proprietary databases that derive from public records of property transactions. They provide information on real-estate transactions. On the other hand, we used scraped data from internet services, which gather advertisements by real-estate agents and private sellers (Leboncoin in France, Fotocasa in Spain): the database also aggregates information on real-estate offers. Such unconventional data is often viewed as interesting proxies to measure, and better understand spatial behaviors and territorial dynamics, and as a means of providing higher spatio-temporal resolution data when compared to institutional data sources  (Gallotti et al.
26 The 10 case studies cover 4 countries of the ESPON Area as well as one cross-border region: Geneva-Annecy-Annemasse (Switzerland-France), Avignon and Paris (France), Madrid, Barcelona and Palma de Majorca (Spain) and Warsaw, Łódź and Krakow (Poland). They have been selected for their variety (size and urban dynamics), data availability, and local expertise on data sources. Because they cover cities with both high and low levels of owner-occupancy, the interest of the selection is to highlight a range of housing regimes in Europe, and also allows to consider how to fill data gaps, as on Figure  2  (Poland, for instance, was poorly documented in European databases). The selected case studies occupy various positions in the urban hierarchy: the spectrum ranges from 3rd tier cities functional urban areas (276,000 inhabitants for Annecy, 320,000 for Avignon or 670,000 for Palma de Majorca), up to larger cities (3.
We also used as source of information a recently published report by  Julliard and Gusarova (2019). Table  1  offers an overview of potential data sources available for surveyed countries. In Poland, a variety of datasets are available: for some cities the transaction price can be obtained through notarial deeds, down to the address (Łódź), but this has not been the case in Warsaw or Krakow. Some datasets are also collected by the National Bank of Poland (LAU2 units). The online real-estate market place domiporta. pl is a widely used resource, that provides real-estate advertisement: it is a harvestable source for price asked and rent asked. Airbnb advertises properties in many cities, but the insideAirBnb harvesting platform is not available. Except.
Harmonization of the database, description of variables Data collection/cleaning and aggregation procedures are conducted so as to produce data that can be used in a multiscale perspective. Produced at the local fine-grain geographical city level (grid and LAU2), they are also aggregated up to the level of the entire FUAs. The indicators created are described below, and summarized Table  3. Local data produced contain the following information, where available: Official geometries identifiers (LAU2 and grid) Income data (municipal and national income data) Aggregated transaction data if available (rooms, surface, price, debt contracted) Aggregated web-scraped data for property sales (rooms, surface, price) Aggregated web-scraped data for property rental (rooms, surface, price) The production of indicators revolved around the premises that price-to-income ratios, and their derivatives, are a commonly agreed characterization of general affordability.
3. income ratio, a measure often considered in the literature  (Kim and Renaud 2009; Tighe and Mueller 2012; Friggit 2017). One of the advantages is how this measure combines housing prices (ownership or rental) as a numerator and a denominator related to wealth or socio-economic situation of the population (income). PIR are easy to understand and convey clear policy data informed messages, such as the number of months of full-time wage required to buy or rent 1 square meter 11. The database delivers 29 targeted indicators at the FUA level, relevant for discussing housing and compare characterization between FUAs. These are provided at LAU2 level. It corresponds to 1390 territorial units in the Paris FUA, 43 in Avignon, 286 in the Geneva cross-border Area, 129 in Barcelona, 137 in Madrid, 33 in Palma de Mallorca, 151 in the 3 Polish FUAs.
Quality control and data validation Two main issues had to be tackled with when analyzing the consistency of data produced. A first one has been the degree to which it was statistically reliable to match institutional and robust census data with scraped and maybe biased datasets. A second issue was how to address missing values, locally poor sampling of spatial data, and spatial biases. This section describes how such issues have been tackled with, and mitigation procedures that have been applied and approved by ESPON EGTC, in the process of publishing the datasets on ESPON Database Portal.
To avoid this lack of comparison between pricing references, we designed our research so as to be able to estimate the quality of scraped data, elaborating on researches that have conducted estimations and quality control on similar data sources. First, 2006 Beresewicz (2015)  specifically addresses the question of comparing the representativeness of Internet Data Sources (IDS) compared to conventional data sources to analyze the real estate market in Poland. Institutional database on transactions allows to compare with harvested big data sources (real estate websites, open data sources and warehouses, Airbnb…). He also evaluates to which extent such online resources can be considered reliable to estimate quantitative information of the housing market. He compares basic variables, such as price per sq.
m, floor area, number of rooms, to verify the consistency of data between institutional data and IDS. This research found the IDS data sources to be mostly consistent over time, except for larger apartments which are under-represented. He finally insists on the fact that "due to the non-sampling character of data obtained from the Internet, it is challenging to estimate standard errors for the estimated characteristics», and that transactions institutional data "do not contain any information on standard errors of estimates, which again limits the scope of comparison of distributions" (2006, p. 54). We obviously faced similar issues. We therefore sought to estimate the differences in datasets by comparing the statistical distribution of those constructed with institutional sources on the one hand, and those gathered by the means of real-estate website harvesting, for case-studies for which both sources were available.
Elaborating on a very common indicator, price per sq. meter. Figure  3  shows that there are significant differences between institutional data and IPS data, especially in Avignon, Paris and Barcelona. Geneva, Annecy and Annemasse ISP data however seem to better fit the distribution of transaction data. We also plot on Figure  3  the distribution of price per sq. meters for case-studies for which only harvested data from ISP: it has not been possible to compare ISP data to other sources in these cases (Madrid, Palma and Warsaw, Łódź and Krakow). 51 To elaborate further on the differences between data sources, and to document whether scraped data should be used assuming they are representative as an alternative data source, we assume that scraped data have in common a non-sampling character.
Recent work also analyzed available data and possible bias and representativeness issues  (Boulay G. , Blanke D. , Casanova Enault L. , and Granié A. , 2020). They showed that transaction data (where available, for instance in France) do not contain any information on standard errors of estimates, which limits the scope of comparison between statistical distributions. They compared, on the one hand, DVF and PERVAL databases, both released as transaction database, the first one being produced for fiscal purposes, the other one as a listing of transactions registered by the Chamber of the Notaries. On both sources, they noted the non-sampling character of data, notaries listing being incomplete (especially outside of the Paris region) 14 , and DVF data being reconstructed as pseudo-transactions after post-processing arbitrages conducted on records that describe parcels (many of them with multiple units, i.
e. several apartments; or any combination of houses, apartments, commercial units). They conclude that fiscal data and notaries data do not exactly convey the same representation of pricing and market segments. More importantly, they insist that public records made available as open-data (Open-DVF) do not allow to analyze the volume of transactions, neither to easily distinguish new housing units, against stock housing units  (Casanova Enault et al. , 2019). Casanova et al. (2017)  importantly conclude that discrepancies observed between DVF and PERVAL derive from "1) the sampling biases induced by the incomplete and unknown coverage of the market in the PERVAL database, 2) differences in choice in the segmentation applied to DVF (postprocessing of multiple transactions in parcel records)"  (Casanova et al.
, 2017, p. 726, Author's translation). 52 To illustrate this point, we analyzed data gathered with different sources for the Avignon FUA (Figure  4 ) and the Paris FUA (Figure  5 ). Both are functional urban areas for which both institutional data (transactions from the notaries) and scraped data (advertisements) were available for the study. Additionally, to better inform the validity of collected data, we used DVF fiscal data (the Etalab version), a dataset that was not available at the time of the study delivered to ESPON. We however used DVF in this paper because records are exhaustive, and are useful to benchmark the quality of scraped data against public records. Figure  4  and Figure  5  show the resulting distribution comparisons (box plots), a Kruskal-Wallis test and a pairwise Dunn test.
Although box-plots demonstrate on average similar distribution patterns of means, median, Q1 and Q3 across the samples compared, the Kruskal-Wallis test demonstrates that distributions significantly differ one from another, and this is especially true with the number of rooms, which seems unreliable in scraped data, compared to other datasets. The pairwise comparison shows that transactions and fiscal data, although different, are also in general more similar in Avignon. In Paris FUA, all databases show significant differences in means, for every variable. Scraped data 2019 and BIEN 2012 however show similar means, and the pairwise test does not allow to reject the null hypothesis. Such a similarity in means and median is counter intuitive, given the tremendous effect of inflation in Paris.
We infer that property markets are discrete social data, similar to Tobler's hypothesis  (Tobler and Wineburg 1971) : a potential price for a specific location is a function of distance to nearby similar transactions, and also a function of the number of properties available, turnover and realized transactions. This method removes spatial bias, resolving the Modifiable Areal Unit Problem (MAUP). We apply Stewart's potential to house price, and also to the number of sellers and buyers of each occupational category, using the SpatialPosition R package  (Commenges and Giraud 2016; Commenges, Giraud, and Lambert 2016). Resulting values have been estimated negative exponential function, with a span of 5 km and a distance decay parameter of -2. An example of the resulting visualization is provided in Figure  6.
Rentals are also more expensive in the French peripheral areas located on the Jura foothills (Pays de Gex: Divonne-les-Bains, Gex, Cessy, Sergy), and also on the Southern border, in the well connected, highly accessible, areas (highways) between France and Switzerland (St-Julien-en-Genevois). Such results from the dataset are counter-intuitive, as the tense situation of the rental market in Geneva city has often been commented. However, a main explanation for relatively "lower" rental prices in Switzerland is due to the institutional protection regarding tenants and rental prices evolution. This leads to another conception of inequalities and housing: where are rental markets the more profitable for landlords (Figure  9b ). One common approach is to calculate how much one landlord should invest for 1 euro of rent.
Supplemental Materials A reproducible example, on the Barcelona case-study, is provided as an HTML RMarkdown document at the following URL (permanent data repository): https:// nakala. fr/10. 34847/nkl. aaea911g 3. This data review has been realized by available team members with specific local expertise. Three additional countries have also been surveyed in the ESPON Project, without delivering the potential for case-study selection : Norway, Sweden and the United Kingdom. 4. DVF has been released at the end of the project funding scheme, and has therefore not been included in the research.
5. As noted by  Julliard and Gusarova (2019)  12. The HousingStat function is delivered as part of the Supplemental Materials, section 2, on the data repository. 13. The unit for these paramaters is the LAU and not the individual level. 14. Representativeness of transaction database have been evaluated in 2002  (David et al. , 2002) , and the authors of the study estimated as 79% in the Paris region in 2001 and varied between 20 and 80% in other cities in France. A decree, passed in 2013, has reinforced the mandatory data transmission from notarial offices, which is now considered as consistent, although incomplete  (Casanova et al. , 2017)  15. A more thorough analysis is currently being conducted under the ANR WIsDHoM research project, with recent extracts of PERVAL and BIEN, and access to the restricted version of DVF data, with results expected in 2021.
ABSTRACTS This data-paper presents and describes a consolidated, harmonized, internationally comparable database to quantify the impacts of the housing affordability crisis. (Poland). We bring insights on how institutional data (i. e. transactions data), can be bridged with unconventional data ("big data" harvested on line) to provide a cost-effective and harmonized data collection effort that can contribute to compare affordability within cities (between neighborhoods) and across cities, using various geographical levels (1km square-grid, municipalities, FUA). We present the structure of the database, how it has been constructed in a reproducible manner; we document the validation process, the strengths and limitations of the data provided, and document the reproducibility of the workflow.
has a long-standing well-established tradition of nationally collecting statistical information on transactions. Given a costly fee (even for researcher of public institutions), transaction data are made available commercially by the Paris Chamber of Notaries database (BIEN). This database covers Ile-de-France real-estate transactions. For the rest of France, the same type of information are commercially made available by French notaries as PERVAL database. Such databases provide a wide range of information related to transactions: price, very detailed characteristics of the property, as well as extra information such as the socio-economic attributes of the seller and the buyer (age, employment category), or the contracted debt of the buyer. Data on property values declared to for land property taxation has been made available in 2019 4.
This fiscal database (Demandes de valeurs foncières, DVF) is available free of charge. It however does not exactly cover transactions on the housing markets, but individual parcels and buildings known values. The problem being that one transactions is in many cases composed by several parcels (i. e. one apartment and one garage; one house, a piece of land, and sometimes a garage and an outbuilding). Major differences between transactions databases and DVF, and their possible use, has been thoroughly documented by Casanova et al. (2017) ; Casanova Enault, Boulay, and Coulon (2019). Furthermore, many online platforms coexist, that provide different entry-points as IDS. We used one of the most popular ones: Leboncoin. fr marketplace advertises properties to buy and properties to let, either by real-estate agents or by individuals5.