elasticsearch/elasticsearch — Issue #1582: Blob storage gateway: deleting an index named x destroys data for any index with name beginning with x
Author: alambert
Created: 2012-01-02 15:08:08+00:00
State: open

Thank you for your help in issue #1564. I am using ElasticSearch v0.18.5 with the S3 shared storage gateway and have been experiencing index data loss: after a restart, some indexes will appear with correct mappings but zero documents.

It looks like this happens because deleting an index named "x" causes ES to remove all keys with prefix
"clustername/indicies/x" which includes data for any index whose name begins with "x". It should only remove keys with prefix "clustername/indicies/x/".

To reproduce:

$ curl -XPUT 'http://localhost:9200/x/'
{"ok":true,"acknowledged":true}
$ curl -XPUT 'http://localhost:9200/x2/'
{"ok":true,"acknowledged":true}
$ curl -XPOST 'http://localhost:9200/x/_gateway/snapshot'
{"ok":true,"_shards":{"total":5,"successful":4,"failed":0}}
$ curl -XPOST 'http://localhost:9200/x2/_gateway/snapshot'
{"ok":true,"_shards":{"total":5,"successful":5,"failed":0}}
$ curl -XDELETE 'http://localhost:9200/x/'
{"ok":true,"acknowledged":true}

After restarting and restoring from the gateway, x2 should have zero documents. After the DELETE request is sent, ES deletes the commit data for index x2:

[2012-01-02 14:53:00,664][INFO ][com.amazonaws.request ] Sending Request: GET http://xxxxxxxx-app-dev-es.s3.amazonaws.com / Parameters: (prefix: es-amalgam-20111025/indices/x, ) Headers: (Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
...
[2012-01-02 14:53:00,954][INFO ][com.amazonaws.request ] Sending Request: DELETE http://xxxxxxxx-app-dev-es.s3.amazonaws.com /es-amalgam-20111025%2Findices%2Fx2%2F4%2Fcommit-0 Headers: (Content-Type: application/x-www-form-urlencoded; charset=utf-8, )

The full logs for the DELETE request are below:

[2012-01-02 14:53:00,405][DEBUG][cluster.service ] [aws-e1b-8.xxxxxxxx.net] processing [delete-index [x]]: execute
[2012-01-02 14:53:00,405][INFO ][cluster.metadata ] [aws-e1b-8.xxxxxxxx.net] [x] deleting index
[2012-01-02 14:53:00,409][DEBUG][cluster.service ] [aws-e1b-8.xxxxxxxx.net] cluster state updated, version [220], source [delete-index [x]]
[2012-01-02 14:53:00,409][DEBUG][river.cluster ] [aws-e1b-8.xxxxxxxx.net] processing [reroute_rivers_node_changed]: execute
[2012-01-02 14:53:00,409][DEBUG][river.cluster ] [aws-e1b-8.xxxxxxxx.net] processing [reroute_rivers_node_changed]: no change in cluster_state
[2012-01-02 14:53:00,410][DEBUG][indices.cluster ] [aws-e1b-8.xxxxxxxx.net] [x] deleting index
[2012-01-02 14:53:00,410][DEBUG][indices ] [aws-e1b-8.xxxxxxxx.net] deleting Index [x]
[2012-01-02 14:53:00,410][DEBUG][index.service ] [aws-e1b-8.xxxxxxxx.net] [x] deleting shard_id [0]
[2012-01-02 14:53:00,410][DEBUG][index.service ] [aws-e1b-8.xxxxxxxx.net] [x] deleting shard_id [1]
[2012-01-02 14:53:00,411][DEBUG][index.service ] [aws-e1b-8.xxxxxxxx.net] [x] deleting shard_id [2]
[2012-01-02 14:53:00,411][DEBUG][index.shard.service ] [aws-e1b-8.xxxxxxxx.net] [x][0] state: [STARTED]->[CLOSED], reason [deleting index]
[2012-01-02 14:53:00,411][DEBUG][index.shard.service ] [aws-e1b-8.xxxxxxxx.net] [x][1] state: [STARTED]->[CLOSED], reason [deleting index]
[2012-01-02 14:53:00,411][DEBUG][index.shard.service ] [aws-e1b-8.xxxxxxxx.net] [x][2] state: [STARTED]->[CLOSED], reason [deleting index]
[2012-01-02 14:53:00,411][INFO ][com.amazonaws.request ] Sending Request: GET http://xxxxxxxx-app-dev-es.s3.amazonaws.com / Parameters: (prefix: es-amalgam-20111025/indices/x/1, ) Headers: (Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
[2012-01-02 14:53:00,412][DEBUG][index.service ] [aws-e1b-8.xxxxxxxx.net] [x] deleting shard_id [3]
[2012-01-02 14:53:00,412][DEBUG][index.service ] [aws-e1b-8.xxxxxxxx.net] [x] deleting shard_id [4]
[2012-01-02 14:53:00,412][DEBUG][index.shard.service ] [aws-e1b-8.xxxxxxxx.net] [x][3] state: [STARTED]->[CLOSED], reason [deleting index]
[2012-01-02 14:53:00,413][INFO ][com.amazonaws.request ] Sending Request: GET http://xxxxxxxx-app-dev-es.s3.amazonaws.com / Parameters: (prefix: es-amalgam-20111025/indices/x/0, ) Headers: (Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
[2012-01-02 14:53:00,413][DEBUG][index.shard.service ] [aws-e1b-8.xxxxxxxx.net] [x][4] state: [STARTED]->[CLOSED], reason [deleting index]
[2012-01-02 14:53:00,414][INFO ][com.amazonaws.request ] Sending Request: GET http://xxxxxxxx-app-dev-es.s3.amazonaws.com / Parameters: (prefix: es-amalgam-20111025/indices/x/2, ) Headers: (Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
[2012-01-02 14:53:00,415][INFO ][com.amazonaws.request ] Sending Request: GET http://xxxxxxxx-app-dev-es.s3.amazonaws.com / Parameters: (prefix: es-amalgam-20111025/indices/x/3, ) Headers: (Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
[2012-01-02 14:53:00,415][INFO ][com.amazonaws.request ] Sending Request: GET http://xxxxxxxx-app-dev-es.s3.amazonaws.com / Parameters: (prefix: es-amalgam-20111025/indices/x/4, ) Headers: (Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
[2012-01-02 14:53:00,432][INFO ][com.amazonaws.request ] Received successful response: 200, AWS Request ID: 1CF9DDBE56ACAE52
[2012-01-02 14:53:00,432][DEBUG][indices.memory ] [aws-e1b-8.xxxxxxxx.net] recalculating shard indexing buffer (reason=removed_shard[x][4]), total is [1gb] with [5] active shards, each shard set to [225.1mb]
[2012-01-02 14:53:00,437][INFO ][com.amazonaws.request ] Received successful response: 200, AWS Request ID: 5439E48D1C6C66E3
[2012-01-02 14:53:00,437][INFO ][com.amazonaws.request ] Sending Request: DELETE http://xxxxxxxx-app-dev-es.s3.amazonaws.com /es-amalgam-20111025%2Findices%2Fx%2F0%2F__0 Headers: (Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
[2012-01-02 14:53:00,456][INFO ][com.amazonaws.request ] Received successful response: 204, AWS Request ID: D5F6AEE4C87331BA
[2012-01-02 14:53:00,456][INFO ][com.amazonaws.request ] Sending Request: DELETE http://xxxxxxxx-app-dev-es.s3.amazonaws.com /es-amalgam-20111025%2Findices%2Fx%2F0%2Fcommit-0 Headers: (Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
[2012-01-02 14:53:00,465][INFO ][com.amazonaws.request ] Received successful response: 200, AWS Request ID: 4C3580B5A6536906
[2012-01-02 14:53:00,465][INFO ][com.amazonaws.request ] Sending Request: DELETE http://xxxxxxxx-app-dev-es.s3.amazonaws.com /es-amalgam-20111025%2Findices%2Fx%2F3%2F__0 Headers: (Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
[2012-01-02 14:53:00,480][INFO ][com.amazonaws.request ] Received successful response: 200, AWS Request ID: E4C5A1CD3FCC0A51
[2012-01-02 14:53:00,480][INFO ][com.amazonaws.request ] Sending Request: DELETE http://xxxxxxxx-app-dev-es.s3.amazonaws.com /es-amalgam-20111025%2Findices%2Fx%2F2%2F__0 Headers: (Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
[2012-01-02 14:53:00,483][INFO ][com.amazonaws.request ] Received successful response: 204, AWS Request ID: BFE115230486752E
[2012-01-02 14:53:00,483][DEBUG][indices.memory ] [aws-e1b-8.xxxxxxxx.net] recalculating shard indexing buffer (reason=removed_shard[x][0]), total is [1gb] with [5] active shards, each shard set to [225.1mb]
[2012-01-02 14:53:00,494][INFO ][com.amazonaws.request ] Received successful response: 204, AWS Request ID: 9E9CE9A22BFF2209
[2012-01-02 14:53:00,494][INFO ][com.amazonaws.request ] Sending Request: DELETE http://xxxxxxxx-app-dev-es.s3.amazonaws.com /es-amalgam-20111025%2Findices%2Fx%2F3%2Fcommit-0 Headers: (Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
[2012-01-02 14:53:00,500][INFO ][com.amazonaws.request ] Received successful response: 204, AWS Request ID: ACAC04C51612FE27
[2012-01-02 14:53:00,500][INFO ][com.amazonaws.request ] Sending Request: DELETE http://xxxxxxxx-app-dev-es.s3.amazonaws.com /es-amalgam-20111025%2Findices%2Fx%2F2%2Fcommit-0 Headers: (Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
[2012-01-02 14:53:00,517][INFO ][com.amazonaws.request ] Received successful response: 204, AWS Request ID: EFBA0DA2BAB7D65E
[2012-01-02 14:53:00,517][DEBUG][indices.memory ] [aws-e1b-8.xxxxxxxx.net] recalculating shard indexing buffer (reason=removed_shard[x][2]), total is [1gb] with [5] active shards, each shard set to [225.1mb]
[2012-01-02 14:53:00,518][INFO ][com.amazonaws.request ] Received successful response: 204, AWS Request ID: DA077754B6C41906
[2012-01-02 14:53:00,518][DEBUG][indices.memory ] [aws-e1b-8.xxxxxxxx.net] recalculating shard indexing buffer (reason=removed_shard[x][3]), total is [1gb] with [5] active shards, each shard set to [225.1mb]
[2012-01-02 14:53:00,626][INFO ][com.amazonaws.request ] Received successful response: 200, AWS Request ID: C97B12628A245B67
[2012-01-02 14:53:00,626][INFO ][com.amazonaws.request ] Sending Request: DELETE http://xxxxxxxx-app-dev-es.s3.amazonaws.com /es-amalgam-20111025%2Findices%2Fx%2F1%2F__0 Headers: (Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
[2012-01-02 14:53:00,643][INFO ][com.amazonaws.request ] Received successful response: 204, AWS Request ID: C444D05B9C2526EC
[2012-01-02 14:53:00,643][INFO ][com.amazonaws.request ] Sending Request: DELETE http://xxxxxxxx-app-dev-es.s3.amazonaws.com /es-amalgam-20111025%2Findices%2Fx%2F1%2Fcommit-0 Headers: (Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
[2012-01-02 14:53:00,662][INFO ][com.amazonaws.request ] Received successful response: 204, AWS Request ID: A2EC73D53CEF11B5
[2012-01-02 14:53:00,663][DEBUG][indices.memory ] [aws-e1b-8.xxxxxxxx.net] recalculating shard indexing buffer (reason=removed_shard[x][1]), total is [1gb] with [5] active shards, each shard set to [225.1mb]
[2012-01-02 14:53:00,664][INFO ][com.amazonaws.request ] Sending Request: GET http://xxxxxxxx-app-dev-es.s3.amazonaws.com / Parameters: (prefix: es-amalgam-20111025/indices/x, ) Headers: (Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
[2012-01-02 14:53:00,701][INFO ][com.amazonaws.request ] Received successful response: 200, AWS Request ID: 126AAAFACB6B9E52
[2012-01-02 14:53:00,701][INFO ][com.amazonaws.request ] Sending Request: DELETE http://xxxxxxxx-app-dev-es.s3.amazonaws.com /es-amalgam-20111025%2Findices%2Fx2%2F0%2F__0 Headers: (Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
[2012-01-02 14:53:00,719][INFO ][com.amazonaws.request ] Received successful response: 204, AWS Request ID: AE11CA882DC3B5F2
[2012-01-02 14:53:00,720][INFO ][com.amazonaws.request ] Sending Request: DELETE http://xxxxxxxx-app-dev-es.s3.amazonaws.com /es-amalgam-20111025%2Findices%2Fx2%2F0%2Fcommit-0 Headers: (Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
[2012-01-02 14:53:00,747][INFO ][com.amazonaws.request ] Received successful response: 204, AWS Request ID: 611EB0181D55BE54
[2012-01-02 14:53:00,747][INFO ][com.amazonaws.request ] Sending Request: DELETE http://xxxxxxxx-app-dev-es.s3.amazonaws.com /es-amalgam-20111025%2Findices%2Fx2%2F1%2F__0 Headers: (Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
[2012-01-02 14:53:00,769][INFO ][com.amazonaws.request ] Received successful response: 204, AWS Request ID: 756ACDD05674B387
[2012-01-02 14:53:00,769][INFO ][com.amazonaws.request ] Sending Request: DELETE http://xxxxxxxx-app-dev-es.s3.amazonaws.com /es-amalgam-20111025%2Findices%2Fx2%2F1%2Fcommit-0 Headers: (Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
[2012-01-02 14:53:00,789][INFO ][com.amazonaws.request ] Received successful response: 204, AWS Request ID: F840A4C6AC34606D
[2012-01-02 14:53:00,789][INFO ][com.amazonaws.request ] Sending Request: DELETE http://xxxxxxxx-app-dev-es.s3.amazonaws.com /es-amalgam-20111025%2Findices%2Fx2%2F2%2F__0 Headers: (Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
[2012-01-02 14:53:00,811][INFO ][com.amazonaws.request ] Received successful response: 204, AWS Request ID: D7CDEDD29EBC7D61
[2012-01-02 14:53:00,812][INFO ][com.amazonaws.request ] Sending Request: DELETE http://xxxxxxxx-app-dev-es.s3.amazonaws.com /es-amalgam-20111025%2Findices%2Fx2%2F2%2Fcommit-0 Headers: (Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
[2012-01-02 14:53:00,833][INFO ][com.amazonaws.request ] Received successful response: 204, AWS Request ID: 9617334FDC66FB23
[2012-01-02 14:53:00,833][INFO ][com.amazonaws.request ] Sending Request: DELETE http://xxxxxxxx-app-dev-es.s3.amazonaws.com /es-amalgam-20111025%2Findices%2Fx2%2F3%2F__0 Headers: (Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
[2012-01-02 14:53:00,904][INFO ][com.amazonaws.request ] Received successful response: 204, AWS Request ID: E7B2AC2C2DFE2ED6
[2012-01-02 14:53:00,904][INFO ][com.amazonaws.request ] Sending Request: DELETE http://xxxxxxxx-app-dev-es.s3.amazonaws.com /es-amalgam-20111025%2Findices%2Fx2%2F3%2Fcommit-0 Headers: (Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
[2012-01-02 14:53:00,926][INFO ][com.amazonaws.request ] Received successful response: 204, AWS Request ID: A47D9D007502212A
[2012-01-02 14:53:00,926][INFO ][com.amazonaws.request ] Sending Request: DELETE http://xxxxxxxx-app-dev-es.s3.amazonaws.com /es-amalgam-20111025%2Findices%2Fx2%2F4%2F__0 Headers: (Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
[2012-01-02 14:53:00,954][INFO ][com.amazonaws.request ] Received successful response: 204, AWS Request ID: 38EC545C47C99D65
[2012-01-02 14:53:00,954][INFO ][com.amazonaws.request ] Sending Request: DELETE http://xxxxxxxx-app-dev-es.s3.amazonaws.com /es-amalgam-20111025%2Findices%2Fx2%2F4%2Fcommit-0 Headers: (Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
[2012-01-02 14:53:00,974][INFO ][com.amazonaws.request ] Received successful response: 204, AWS Request ID: A709CE7D578AAF0B
[2012-01-02 14:53:00,981][DEBUG][gateway.s3 ] [aws-e1b-8.xxxxxxxx.net] writing to gateway org.elasticsearch.gateway.shared.SharedStorageGateway$2@3d761403 ...
[2012-01-02 14:53:00,984][DEBUG][cluster.service ] [aws-e1b-8.xxxxxxxx.net] processing [delete-index [x]]: done applying updated cluster_state
[2012-01-02 14:53:00,984][INFO ][com.amazonaws.request ] Sending Request: PUT http://xxxxxxxx-app-dev-es.s3.amazonaws.com /es-amalgam-20111025%2Fmetadata%2Fmetadata-2455 Headers: (Content-Length: 77420, Content-Type: application/octet-stream, )
[2012-01-02 14:53:01,057][INFO ][com.amazonaws.request ] Received successful response: 200, AWS Request ID: BAD343B536EDE0F7
[2012-01-02 14:53:01,057][INFO ][com.amazonaws.request ] Sending Request: GET http://xxxxxxxx-app-dev-es.s3.amazonaws.com / Parameters: (prefix: es-amalgam-20111025/metadata/, ) Headers: (Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
[2012-01-02 14:53:01,105][INFO ][com.amazonaws.request ] Received successful response: 200, AWS Request ID: 34F9E73651ADB6CD
[2012-01-02 14:53:01,105][INFO ][com.amazonaws.request ] Sending Request: DELETE http://xxxxxxxx-app-dev-es.s3.amazonaws.com /es-amalgam-20111025%2Fmetadata%2Fmetadata-2454 Headers: (Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
[2012-01-02 14:53:01,131][INFO ][com.amazonaws.request ] Received successful response: 204, AWS Request ID: 75732598971E7A5F
[2012-01-02 14:53:01,131][DEBUG][gateway.s3 ] [aws-e1b-8.xxxxxxxx.net] wrote to gateway org.elasticsearch.gateway.shared.SharedStorageGateway$2@3d761403, took 150ms

I can provide additional logs if needed. Thank you!