brianchandotcom/liferay-portal — pull request #22321 by shuyangzhou (2014-11-19 01:15:40+00:00)
Title: LPS-51661 Bump up elasticsearch cluster health request timeout time to 1hour for test

CC @mhan810 @michaelhashimoto

I finally managed to reproduce this.
I changed BackupAndRestoreIndexesTest to run the testBackupAndRestore in an infinite loop.
Then starts up 8 terminals to run "while true; do true; done" to fully consume my cpus, after a couple loop in the test, I got:

[junit] Unable to initialize Elasticsearch cluster: {
[junit] "cluster_name" : "LiferayElasticSearch",
[junit] "status" : "red",
[junit] "timed_out" : true,
[junit] "number_of_nodes" : 1,
[junit] "number_of_data_nodes" : 1,
[junit] "active_primary_shards" : 15,
[junit] "active_shards" : 15,
[junit] "relocating_shards" : 0,
[junit] "initializing_shards" : 1,
[junit] "unassigned_shards" : 0
[junit] }
[junit] java.lang.IllegalStateException: Unable to initialize Elasticsearch cluster: {
[junit] "cluster_name" : "LiferayElasticSearch",
[junit] "status" : "red",
[junit] "timed_out" : true,
[junit] "number_of_nodes" : 1,
[junit] "number_of_data_nodes" : 1,
[junit] "active_primary_shards" : 15,
[junit] "active_shards" : 15,
[junit] "relocating_shards" : 0,
[junit] "initializing_shards" : 1,
[junit] "unassigned_shards" : 0
[junit] }
[junit]	at com.liferay.portal.search.elasticsearch.ElasticsearchSearchEngine.restore(ElasticsearchSearchEngine.java:230)
[junit]	at com.liferay.portal.kernel.search.SearchEngineProxyWrapper.restore(SearchEngineProxyWrapper.java:115)
[junit]	at com.liferay.portal.kernel.search.SearchEngineUtil.restore(SearchEngineUtil.java:670)

Same failure as the broken tests:
https://test-14-2.liferay.com/job/test-portal-pullrequest-backend%28master%29/332/
http://test-1-2/job/test-portal-pullrequest-backend%28master%29/18/group=23,label_exp=!test-1-2/#showFailuresLink

I don't want to change it to wait for Long.MAX_VALUE, I think 1 hour should be long enough, after all we won''t want CI servers to hang any longer than that