CircleCI 2.0 移行時に直面した問題とその解決方法 for Rails Project

2018年8月31日を最後に、CircleCI 1.0が使えなくなることがアナウンスされました。

See: [CircleCI 1.0 End of Life on August 31, 2018](https://circleci.com/blog/sunsetting-1-0/)

社内向けに展開していたのですが、需要がありそうなので公開してみます。

# ハマったポイント
## 1. rspec_junit_formatter がロードできないと言われる

```
$ #!/bin/bash -eo pipefail
bundle exec rspec --profile 10 \
                  --format RspecJunitFormatter \
                  --format progress \
                  --out test_results/rspec.xml \
                  $(circleci tests glob "spec/**/*_spec.rb" | circleci tests split --split-by=timings)

bundler: failed to load command: rspec (/home/circleci/sbj-os/vendor/bundle/ruby/2.4.0/bin/rspec)
LoadError: cannot load such file -- rspec_junit_formatter
```

### 対応
Gemfile に [sj26/rspec_junit_formatter](https://github.com/sj26/rspec_junit_formatter) を追加する。

公式ドキュメントを読めばちゃんと書いてあるのだけれど、CircleCI の設定を変える時に Gemfile を触るイメージがないのでハマりやすい。

> If they succeed, it stores the test results using store_test_results so we can quickly see our build failures in the Test Summary section. This is why we added RspecJunitFormatter to our Gemfile.
> [Language Guide: Ruby - CircleCI](https://circleci.com/docs/2.0/language-ruby/#config-walkthrough)


## 2. RSpecを並列で実施すると死ぬ
```
$ #!/bin/bash -eo pipefail
mkdir /tmp/test-results
TEST_FILES="$(circleci tests glob "spec/**/*_spec.rb" | circleci tests split --split-by=timings)"
bundle exec rspec --format progress \
                  --format RspecJunitFormatter \
                  --out /tmp/test-results/rspec.xml \
                  "${TEST_FILES}"

bundler: failed to load command: rspec (/home/circleci/sbj_os/vendor/bundle/ruby/2.4.0/bin/rspec)
NoMethodError: undefined method `captures' for nil:NilClass
  /home/circleci/sbj_os/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:124:in `parse_id'
  /home/circleci/sbj_os/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/configuration.rb:1953:in `extract_location'
  /home/circleci/sbj_os/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/configuration.rb:1892:in `block in get_files_to_run'
  /home/circleci/sbj_os/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/flat_map.rb:7:in `each'
  /home/circleci/sbj_os/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/flat_map.rb:7:in `flat_map'
  /home/circleci/sbj_os/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/flat_map.rb:7:in `flat_map'
  /home/circleci/sbj_os/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/configuration.rb:1890:in `get_files_to_run'
  /home/circleci/sbj_os/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/configuration.rb:928:in `files_to_run'
  /home/circleci/sbj_os/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/configuration.rb:1433:in `load_spec_files'
  /home/circleci/sbj_os/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:100:in `setup'
  /home/circleci/sbj_os/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:86:in `run'
  /home/circleci/sbj_os/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:71:in `run'
  /home/circleci/sbj_os/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:45:in `invoke'
  /home/circleci/sbj_os/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/exe/rspec:4:in `<top (required)>'
  /home/circleci/sbj_os/vendor/bundle/ruby/2.4.0/bin/rspec:23:in `load'
  /home/circleci/sbj_os/vendor/bundle/ruby/2.4.0/bin/rspec:23:in `<top (required)>'
```

### 対応
[10/11 追記]
調べた結果 [Spec fails only on CI (details included) · Issue #2451 · rspec/rspec-core](https://github.com/rspec/rspec-core/issues/2451#issuecomment-318797132)
 に辿り着いた。コメントにある通り、下記の記述を削除することで対応した。

```
$(circleci tests glob "spec/**/*_spec.rb" | circleci tests split --split-by=timings)
```

[12/22 追記]
公式ドキュメントのサンプルが修正されていた。下記の通り、shell 実行するように書き換えることで並列実行に成功。
[Language Guide: Ruby - CircleCI](https://circleci.com/docs/2.0/language-ruby/#sample-configuration)

```diff
- - run:
-     name: run tests
-     command: |
-       mkdir /tmp/test-results
-       TEST_FILES="$(circleci tests glob "spec/**/*_spec.rb" | circleci tests split --split-by=timings)"
-       bundle exec rspec --format progress \
-                         --format RspecJunitFormatter \
-                         --out /tmp/test-results/rspec.xml
+ - type: shell
+   command: |
+     bundle exec rspec --profile 10 \
+                       --format RspecJunitFormatter \
+                       --out test_results/rspec.xml \
+                       --format progress \
+                       $(circleci tests glob "spec/**/*_spec.rb" | circleci tests split --split-by=timings)
```

## 3. Capybaraさんが特定のボタンを見つけられず、マウス操作に失敗する

```
Failure/Error: click_link 'カートに入れる'

Capybara::Poltergeist::MouseEventFailed:
  Firing a click at co-ordinates [990, 329.5] failed. Poltergeist detected another element with CSS selector 'html body.forSubdomain div.mainContents article div.container-fluid.os-container.os-sp-container div#dev-brand-site table.table.table-bordered.products tbody tr td' at this position. It may be overlapping the element you are trying to interact with. If you don't care about overlapping elements, try using node.trigger('click').
# spec/steps/order_steps.rb:69:in `block (2 levels) in <top (required)>'
# spec/steps/order_steps.rb:67:in `block in <top (required)>'
# spec/steps/order_steps.rb:63:in `block in <top (required)>'
# ./spec/features/cart.feature:10:in `run_step'
# ./spec/features/cart.feature:9:in `block (4 levels) in run_feature'
# ./spec/features/cart.feature:8:in `each'
# ./spec/features/cart.feature:8:in `block (3 levels) in run_feature'
# ./spec/features/cart.feature:10:in `カートに商品を投入する'
# ------------------
# --- Caused by: ---
# Capybara::Poltergeist::BrowserError:
#   There was an error inside the PhantomJS portion of Poltergeist. If this is the error returned, and not the cause of a more detailed error response, this is probably a bug, so please report it. 
#   
#   Poltergeist.MouseEventFailed: click
#   html body.forSubdomain div.mainContents article div.container-fluid.os-container.os-sp-container div#dev-brand-site table.table.table-bordered.products tbody tr td
#   {"x"=>990, "y"=>329.5}
#   spec/steps/order_steps.rb:69:in `block (2 levels) in <top (required)>'
```

### 対応
[CircleCI上でCapybara, Poltergeist, PhantomJSを利用したテストがMouseEventFailed失敗す… - Sooey](http://journal.sooey.com/269)

## 4. 数回に一回、Capybaraさんが特定文言を見つけられない

```
Failure/Error: find('.alert-info', text: "added!", wait: 30)

Capybara::ElementNotFound:
  Unable to find css ".alert-info" with text "added!"
# spec/steps/order_steps.rb:72:in `block in <top (required)>'
# spec/steps/order_steps.rb:63:in `block in <top (required)>'
# ./spec/features/order.feature:750:in `run_step'
# ./spec/features/order.feature:749:in `block (4 levels) in run_feature'
# ./spec/features/order.feature:748:in `each'
# ./spec/features/order.feature:748:in `block (3 levels) in run_feature'
# ./spec/features/order.feature:750:in `カートに商品を投入する'
```

### 対応
[NoRedInk/rspec-retry](https://github.com/NoRedInk/rspec-retry) で retry するようにした。しばらくこれで様子を見る。

この話題に関しては別に記事を書いているので、興味があればこちらも参照して欲しい。

See: [Capybara::ElementNotFound 回避に wait させるか retry させるか - Qiita](https://qiita.com/tetsuya/items/ac41c53e9e9843ec0dce)