# FieldData Filter

FieldData is an in-memory representation of the term dictionary in an uninverted form. Under certain circumstances this FieldData representation can grow very large on high-cardinality fields like tokenized full-text. Depending on the use-case filtering the terms that are hold in the FieldData representation can heavily improve execution performance and application stability.
FieldData Filters can be applied on a per-segment basis. During FieldData loading the terms enumeration is passed through a filter predicate that  either accepts or rejects a term.

## Frequency Filter

The Frequency Filter acts as a high / low pass filter based on the document frequencies of a certain term within the segment that is loaded into field data. It allows to reject terms that are very high or low frequent based on absolute frequencies or percentages relative to the number of documents in the segment or more precise the number of document that have at least one value in the field that is loaded in the current segment.

Here is an example mapping

Here is an example mapping:

```json
{
    "tweet" : {
        "properties" : {
            "locale" : {
                "type" : "string",
                "fielddata" : "format=paged_bytes;filter.frequency.min=0.001;filter.frequency.max=0.1",
                "index" : "analyzed",
            }
        }
    }
}
```
### Paramters

 * `filter.frequency.min` - the minimum document frequency (inclusive) in order to be loaded in to memory. Either a percentage if < `1.0` or an absolute value. `0` if omitted.
 * `filter.frequency.max` - the maximum document frequency (inclusive) in order to be loaded in to memory. Either a percentage if < `1.0` or an absolute value. `0` if omitted.
 * `filter.frequency.min_segment_size` - the minimum number of documents in a segment in order for the filter to be applied. Small segments might be omitted with this setting.

## Regular Expression Filter

The regular expression filter applies a regular expression to each term  during loading and only loads terms into memory that match the given regular expression.

Here is an example mapping:

```json
{
    "tweet" : {
        "properties" : {
            "locale" : {
                "type" : "string",
                "fielddata" : "format=paged_bytes;filter.regex=^en_.*",
                "index" : "analyzed",
            }
        }
    }
}
```

Closes #2874
Configure FieldData using a hash not a string

Closes #2876
Fix term counters in script field terms facet

Fixes #2878
upgrade to netty 3.6.5
Fix for extended facets test.
Should prevent OOM
Better error description if field(s) (statistical facet) and value_field (term_stats facet) are not a numeric field
Ordinals were omitted from fielddata cache size calculation if field has more than one term.
Closes #2882
rename prefix_length to prefix_len for consistency

Closes #2883
Test that size is actually computed.
Relates to #2882
Make sure that settings are propagated to all nodes
If no options are specified with the clear cache api then all caches should be cleared.
Closes #2886
fix memory computation for int field data
Field data: Try to load short values as byte values and load int values as short or byte values to reduce the size they take in memory.
CacheTests test fix.
Add toString() for FilterBuilders
Closes #2887.
Expose field level field data statistics
closes #2889
Clear Cache API: Streamline option names
closes #2890
Add more dynamic settings validation
Optimization in fielddata cache where ordinals are used instead of flat arrays when number of unique values is low