elasticsearch/elasticsearch.github.com — Issue #134: Adding large numbers of Types makes server unresponsive
Author: gibrown
Created: 2012-01-06 18:49:59+00:00
State: open

Trying to add a large number of Types (Put Mapping) to an ElasticSearch cluster with a single server while also indexing data into the index causes the server to become unresponsive presumably due to the build up of processing the new Mappings. The unresponsiveness affects both the index being added to as well as any other indices on the server. After some (long) amount of processing time all of the indices do become responsive.

[Status of server after becoming responsive again.](https://gist.github.com/1571828) Not sure how to read all of the stats, but the 54 min index time seems to be a problem. The server was starting to be unresponsive after adding about 200k documents (each one is pretty short text), and the indexing script (using Elastica) timed out trying to access the server after 530k docs.

Bulk Indexing script:

```php

$s = 0;
$c = 0;
$types = array();
while ( ( $rows = $wpdb->get_results( $wpdb->prepare( "SELECT * FROM other ORDER BY other_id ASC LIMIT %d, 100", $s ) ) ) ) {
echo date( 'H:i:s' ) . " working on 100 records from $s\n";
set_time_limit( 360 );
$docs = array();
foreach( $rows as $row ) {
$es_key = "pd_" . $row->pd_id;
if ( false == isset( $types[ $row->pd_id ] ) ) {
echo "*";
$type = new Elastica_Type( $index, $es_key );
$mapping = new Elastica_Type_Mapping(
$type,
array(
'date' => array( 'type' => 'date', 'store' => 'yes', 'index' => 'not_analyzed', 'format' => 'yyyy-mm-dd hh:mm:ss||yyyy-mm-dd' ),
'q1' => array( 'type' => 'string', 'store' => 'yes', 'index' => 'analyzed', 'analyzer' => 'analyzer_pd' ),
'q1_string' => array( 'type' => 'string', 'store' => 'yes', 'index' => 'not_analyzed' )
) );
$type->setMapping( $mapping );
$types[ $row->pd_id ] = 1;
} else {
echo "-";
}
}
sleep( 1 );
reset( $rows );
foreach( $rows as $row ) {
$es_key = "pd_" . $row->pd_id;
$docs[] = new Elastica_Document( $row->other_id, array( 'date' => date( 'Y-m-d' ), 'q1' => $row->other_text, 'q1_string' => $row->other_text ), $es_key, ES_INDEX_NAME );
$c = $row->other_id;
echo "!";
}
es_index_docs( $docs );
$s += 100;
echo "\nother_id: $c\n";
sleep( 1 );
}
$client->getIndex( ES_INDEX_NAME )->refresh();
$client->getIndex( ES_INDEX_NAME )->optimize();
```

[Original Background](https://groups.google.com/group/elasticsearch/browse_thread/thread/d309372c31774a4a/a2a8d2ee37853288?lnk=gst&q=Greg#a2a8d2ee37853288)

ElasticSearch version: 0.18.5