Project: mesos
Issue MESOS-5812: MasterAPITest.Subscribe is flaky
This test seems to be flaky, although on Mac OS X and CentOS 7 the error a bit different. On Mac OS X: {noformat}[ RUN ] ContentType/MasterAPITest.Subscribe/0 I0708 11:42:48.474665 1927435008 cluster.cpp:155] Creating default 'local' authorizer I0708 11:42:48.480677 1927435008 leveldb.cpp:174] Opened db in 5727us I0708 11:42:48.481494 1927435008 leveldb.cpp:181] Compacted db in 722us I0708 11:42:48.481541 1927435008 leveldb.cpp:196] Created db iterator in 19us I0708 11:42:48.481572 1927435008 leveldb.cpp:202] Seeked to beginning of db in 9us I0708 11:42:48.481587 1927435008 leveldb.cpp:271] Iterated through 0 keys in the db in 7us I0708 11:42:48.481617 1927435008 replica.cpp:779] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I0708 11:42:48.482030 350982144 recover.cpp:451] Starting replica recovery I0708 11:42:48.482203 350982144 recover.cpp:477] Replica is in EMPTY status I0708 11:42:48.484107 348299264 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (3780)@127.0.0.1:50325 I0708 11:42:48.484318 350982144 recover.cpp:197] Received a recover response from a replica in EMPTY status I0708 11:42:48.484750 348835840 master.cpp:382] Master e055d60c-05ff-487e-82da-d0a43e52605c (localhost) started on 127.0.0.1:50325 I0708 11:42:48.484850 349908992 recover.cpp:568] Updating replica status to STARTING I0708 11:42:48.484788 348835840 master.cpp:384] Flags at startup: --acls="" --agent_ping_timeout="15secs" --agent_reregister_timeout="10mins" --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate_agents="true" --authenticate_frameworks="true" --authenticate_http="true" --authenticate_http_frameworks="true" --authenticators="crammd5" --authorizers="local" --credentials="/private/tmp/Sn2Kf4/credentials" --framework_sorter="drf" --help="false" --hostname_lookup="true" --http_authenticators="basic" --http_framework_authenticators="basic" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_agent_ping_timeouts="5" --max_completed_frameworks="50" --max_completed_tasks_per_framework="1000" --quiet="false" --recovery_agent_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_store_timeout="100secs" --registry_strict="true" --root_submissions="true" --user_sorter="drf" --version="false" --webui_dir="/usr/local/share/mesos/webui" --work_dir="/private/tmp/Sn2Kf4/master" --zk_session_timeout="10secs" W0708 11:42:48.485263 348835840 master.cpp:387] ************************************************** Master bound to loopback interface! Cannot communicate with remote schedulers or agents. You might want to set '--ip' flag to a routable IP address. ************************************************** I0708 11:42:48.485291 348835840 master.cpp:434] Master only allowing authenticated frameworks to register I0708 11:42:48.485314 348835840 master.cpp:448] Master only allowing authenticated agents to register I0708 11:42:48.485335 348835840 master.cpp:461] Master only allowing authenticated HTTP frameworks to register I0708 11:42:48.485347 348835840 credentials.hpp:37] Loading credentials for authentication from '/private/tmp/Sn2Kf4/credentials' I0708 11:42:48.485373 349372416 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 397us I0708 11:42:48.485414 349372416 replica.cpp:320] Persisted replica status to STARTING I0708 11:42:48.485608 350982144 recover.cpp:477] Replica is in STARTING status I0708 11:42:48.485749 348835840 master.cpp:506] Using default 'crammd5' authenticator I0708 11:42:48.485852 348835840 master.cpp:578] Using default 'basic' HTTP authenticator I0708 11:42:48.486018 348835840 master.cpp:658] Using default 'basic' HTTP framework authenticator I0708 11:42:48.486140 348835840 master.cpp:705] Authorization enabled I0708 11:42:48.486486 350982144 replica.cpp:673] Replica in STARTING status received a broadcasted recover request from (3783)@127.0.0.1:50325 I0708 11:42:48.486758 352055296 recover.cpp:197] Received a recover response from a replica in STARTING status I0708 11:42:48.487176 350982144 recover.cpp:568] Updating replica status to VOTING I0708 11:42:48.487576 352055296 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 300us I0708 11:42:48.487658 352055296 replica.cpp:320] Persisted replica status to VOTING I0708 11:42:48.487736 350982144 recover.cpp:582] Successfully joined the Paxos group I0708 11:42:48.487951 350982144 recover.cpp:466] Recover process terminated I0708 11:42:48.489441 348835840 master.cpp:1973] The newly elected leader is master@127.0.0.1:50325 with id e055d60c-05ff-487e-82da-d0a43e52605c I0708 11:42:48.489518 348835840 master.cpp:1986] Elected as the leading master! I0708 11:42:48.489545 348835840 master.cpp:1673] Recovering from registrar I0708 11:42:48.489637 350982144 registrar.cpp:332] Recovering registrar I0708 11:42:48.490120 351518720 log.cpp:553] Attempting to start the writer I0708 11:42:48.491161 350445568 replica.cpp:493] Replica received implicit promise request from (3784)@127.0.0.1:50325 with proposal 1 I0708 11:42:48.491461 350445568 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 252us I0708 11:42:48.491528 350445568 replica.cpp:342] Persisted promised to 1 I0708 11:42:48.492337 348299264 coordinator.cpp:238] Coordinator attempting to fill missing positions I0708 11:42:48.493482 349372416 replica.cpp:388] Replica received explicit promise request from (3785)@127.0.0.1:50325 for position 0 with proposal 2 I0708 11:42:48.493854 349372416 leveldb.cpp:341] Persisting action (8 bytes) to leveldb took 283us I0708 11:42:48.493904 349372416 replica.cpp:712] Persisted action at 0 I0708 11:42:48.495302 348299264 replica.cpp:537] Replica received write request for position 0 from (3786)@127.0.0.1:50325 I0708 11:42:48.495455 348299264 leveldb.cpp:436] Reading position from leveldb took 45us I0708 11:42:48.495761 348299264 leveldb.cpp:341] Persisting action (14 bytes) to leveldb took 261us I0708 11:42:48.495803 348299264 replica.cpp:712] Persisted action at 0 I0708 11:42:48.496484 350445568 replica.cpp:691] Replica received learned notice for position 0 from @0.0.0.0:0 I0708 11:42:48.496795 350445568 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 255us I0708 11:42:48.496857 350445568 replica.cpp:712] Persisted action at 0 I0708 11:42:48.496896 350445568 replica.cpp:697] Replica learned NOP action at position 0 I0708 11:42:48.497445 350982144 log.cpp:569] Writer started with ending position 0 I0708 11:42:48.498523 350982144 leveldb.cpp:436] Reading position from leveldb took 80us I0708 11:42:48.499307 349908992 registrar.cpp:365] Successfully fetched the registry (0B) in 9.63712ms I0708 11:42:48.499464 349908992 registrar.cpp:464] Applied 1 operations in 36us; attempting to update the 'registry' I0708 11:42:48.499953 351518720 log.cpp:577] Attempting to append 159 bytes to the log I0708 11:42:48.500088 350982144 coordinator.cpp:348] Coordinator attempting to write APPEND action at position 1 I0708 11:42:48.500880 348299264 replica.cpp:537] Replica received write request for position 1 from (3787)@127.0.0.1:50325 I0708 11:42:48.501186 348299264 leveldb.cpp:341] Persisting action (178 bytes) to leveldb took 259us I0708 11:42:48.501231 348299264 replica.cpp:712] Persisted action at 1 I0708 11:42:48.501786 351518720 replica.cpp:691] Replica received learned notice for position 1 from @0.0.0.0:0 I0708 11:42:48.502118 351518720 leveldb.cpp:341] Persisting action (180 bytes) to leveldb took 311us I0708 11:42:48.502260 351518720 replica.cpp:712] Persisted action at 1 I0708 11:42:48.502305 351518720 replica.cpp:697] Replica learned APPEND action at position 1 I0708 11:42:48.503475 349908992 registrar.cpp:509] Successfully updated the 'registry' in 3.944192ms I0708 11:42:48.503909 349908992 registrar.cpp:395] Successfully recovered registrar I0708 11:42:48.504003 350982144 log.cpp:596] Attempting to truncate the log to 1 I0708 11:42:48.504250 349372416 coordinator.cpp:348] Coordinator attempting to write TRUNCATE action at position 2 I0708 11:42:48.504546 350445568 master.cpp:1781] Recovered 0 agents from the Registry (121B) ; allowing 10mins for agents to re-register I0708 11:42:48.506022 352055296 replica.cpp:537] Replica received write request for position 2 from (3788)@127.0.0.1:50325 I0708 11:42:48.506479 352055296 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 320us I0708 11:42:48.506513 352055296 replica.cpp:712] Persisted action at 2 I0708 11:42:48.506978 351518720 replica.cpp:691] Replica received learned notice for position 2 from @0.0.0.0:0 I0708 11:42:48.507155 351518720 leveldb.cpp:341] Persisting action (18 bytes) to leveldb took 169us I0708 11:42:48.507237 351518720 leveldb.cpp:399] Deleting ~1 keys from leveldb took 37us I0708 11:42:48.507264 351518720 replica.cpp:712] Persisted action at 2 I0708 11:42:48.507285 351518720 replica.cpp:697] Replica learned TRUNCATE action at position 2 I0708 11:42:48.521363 1927435008 cluster.cpp:432] Creating default 'local' authorizer I0708 11:42:48.522498 350982144 slave.cpp:205] Agent started on 119)@127.0.0.1:50325 I0708 11:42:48.522538 350982144 slave.cpp:206] Flags at startup: --acls="" --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http="true" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authorizer="local" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/var/folders/ny/tcvyblqj43s2gdh2_895v9nw0000gp/T/ContentType_MasterAPITest_Subscribe_0_VaPndX/credential" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/var/folders/ny/tcvyblqj43s2gdh2_895v9nw0000gp/T/ContentType_MasterAPITest_Subscribe_0_VaPndX/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --http_authenticators="basic" --http_command_executor="false" --http_credentials="/var/folders/ny/tcvyblqj43s2gdh2_895v9nw0000gp/T/ContentType_MasterAPITest_Subscribe_0_VaPndX/http_credentials" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/Users/zhitao/Uber/sync/zhitao-mesos1.dev.uber.com/home/uber/mesos/build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resources="cpus:2;gpus:0;mem:1024;disk:1024;ports:[31000-32000]" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --version="false" --work_dir="/var/folders/ny/tcvyblqj43s2gdh2_895v9nw0000gp/T/ContentType_MasterAPITest_Subscribe_0_VaPndX" W0708 11:42:48.522903 350982144 slave.cpp:209] ************************************************** Agent bound to loopback interface! Cannot communicate with remote master(s). You might want to set '--ip' flag to a routable IP address. ************************************************** I0708 11:42:48.522922 350982144 credentials.hpp:86] Loading credential for authentication from '/var/folders/ny/tcvyblqj43s2gdh2_895v9nw0000gp/T/ContentType_MasterAPITest_Subscribe_0_VaPndX/credential' W0708 11:42:48.522965 1927435008 scheduler.cpp:157] ************************************************** Scheduler driver bound to loopback interface! Cannot communicate with remote master(s). You might want to set 'LIBPROCESS_IP' environment variable to use a routable IP address. ************************************************** I0708 11:42:48.522992 1927435008 scheduler.cpp:172] Version: 1.0.0 I0708 11:42:48.523066 350982144 slave.cpp:343] Agent using credential for: test-principal I0708 11:42:48.523092 350982144 credentials.hpp:37] Loading credentials for authentication from '/var/folders/ny/tcvyblqj43s2gdh2_895v9nw0000gp/T/ContentType_MasterAPITest_Subscribe_0_VaPndX/http_credentials' I0708 11:42:48.523334 350982144 slave.cpp:395] Using default 'basic' HTTP authenticator I0708 11:42:48.523973 352055296 scheduler.cpp:461] New master detected at master@127.0.0.1:50325 I0708 11:42:48.524050 350982144 slave.cpp:594] Agent resources: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0708 11:42:48.524196 350982144 slave.cpp:602] Agent attributes: [ ] I0708 11:42:48.524224 350982144 slave.cpp:607] Agent hostname: localhost I0708 11:42:48.525522 350445568 state.cpp:57] Recovering state from '/var/folders/ny/tcvyblqj43s2gdh2_895v9nw0000gp/T/ContentType_MasterAPITest_Subscribe_0_VaPndX/meta' I0708 11:42:48.525853 350445568 status_update_manager.cpp:200] Recovering status update manager I0708 11:42:48.526165 350445568 slave.cpp:4856] Finished recovery I0708 11:42:48.527223 349372416 status_update_manager.cpp:174] Pausing sending status updates I0708 11:42:48.527231 352055296 slave.cpp:969] New master detected at master@127.0.0.1:50325 I0708 11:42:48.527276 352055296 slave.cpp:1028] Authenticating with master master@127.0.0.1:50325 I0708 11:42:48.527328 352055296 slave.cpp:1039] Using default CRAM-MD5 authenticatee I0708 11:42:48.527561 352055296 slave.cpp:1001] Detecting new master I0708 11:42:48.527582 348299264 authenticatee.cpp:121] Creating new client SASL connection I0708 11:42:48.528666 349908992 master.cpp:6006] Authenticating slave(119)@127.0.0.1:50325 I0708 11:42:48.528880 352055296 authenticator.cpp:98] Creating new server SASL connection I0708 11:42:48.529089 350445568 http.cpp:381] HTTP POST for /master/api/v1/scheduler from 127.0.0.1:50918 I0708 11:42:48.529233 350445568 master.cpp:2272] Received subscription request for HTTP framework 'default' I0708 11:42:48.529261 350445568 master.cpp:2012] Authorizing framework principal 'test-principal' to receive offers for role '*' I0708 11:42:48.529323 352055296 authenticatee.cpp:213] Received SASL authentication mechanisms: CRAM-MD5 I0708 11:42:48.529357 352055296 authenticatee.cpp:239] Attempting to authenticate with mechanism 'CRAM-MD5' I0708 11:42:48.529417 352055296 authenticator.cpp:204] Received SASL authentication start I0708 11:42:48.529503 352055296 authenticator.cpp:326] Authentication requires more steps I0708 11:42:48.529561 352055296 master.cpp:2370] Subscribing framework 'default' with checkpointing disabled and capabilities [ ] I0708 11:42:48.529721 349908992 authenticatee.cpp:259] Received SASL authentication step I0708 11:42:48.530005 348835840 authenticator.cpp:232] Received SASL authentication step I0708 11:42:48.530241 348835840 authenticator.cpp:318] Authentication success I0708 11:42:48.530254 350445568 hierarchical.cpp:271] Added framework e055d60c-05ff-487e-82da-d0a43e52605c-0000 I0708 11:42:48.530900 349908992 authenticatee.cpp:299] Authentication success I0708 11:42:48.531186 350982144 master.cpp:6036] Successfully authenticated principal 'test-principal' at slave(119)@127.0.0.1:50325 I0708 11:42:48.531657 348299264 slave.cpp:1123] Successfully authenticated with master master@127.0.0.1:50325 I0708 11:42:48.531935 349372416 master.cpp:4676] Registering agent at slave(119)@127.0.0.1:50325 (localhost) with id e055d60c-05ff-487e-82da-d0a43e52605c-S0 I0708 11:42:48.532304 349908992 registrar.cpp:464] Applied 1 operations in 55us; attempting to update the 'registry' I0708 11:42:48.532908 348835840 log.cpp:577] Attempting to append 326 bytes to the log I0708 11:42:48.533015 352055296 coordinator.cpp:348] Coordinator attempting to write APPEND action at position 3 I0708 11:42:48.533641 349372416 replica.cpp:537] Replica received write request for position 3 from (3798)@127.0.0.1:50325 I0708 11:42:48.533867 349372416 leveldb.cpp:341] Persisting action (345 bytes) to leveldb took 186us I0708 11:42:48.533917 349372416 replica.cpp:712] Persisted action at 3 I0708 11:42:48.537066 349908992 replica.cpp:691] Replica received learned notice for position 3 from @0.0.0.0:0 I0708 11:42:48.538169 349908992 leveldb.cpp:341] Persisting action (347 bytes) to leveldb took 914us I0708 11:42:48.538226 349908992 replica.cpp:712] Persisted action at 3 I0708 11:42:48.538255 349908992 replica.cpp:697] Replica learned APPEND action at position 3 I0708 11:42:48.539247 352055296 registrar.cpp:509] Successfully updated the 'registry' in 6.895104ms I0708 11:42:48.539302 348299264 log.cpp:596] Attempting to truncate the log to 3 I0708 11:42:48.539393 348299264 coordinator.cpp:348] Coordinator attempting to write TRUNCATE action at position 4 I0708 11:42:48.539798 348835840 master.cpp:4745] Registered agent e055d60c-05ff-487e-82da-d0a43e52605c-S0 at slave(119)@127.0.0.1:50325 (localhost) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0708 11:42:48.539881 348299264 hierarchical.cpp:478] Added agent e055d60c-05ff-487e-82da-d0a43e52605c-S0 (localhost) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (allocated: ) I0708 11:42:48.539901 349908992 slave.cpp:1169] Registered with master master@127.0.0.1:50325; given agent ID e055d60c-05ff-487e-82da-d0a43e52605c-S0 I0708 11:42:48.540287 350445568 status_update_manager.cpp:181] Resuming sending status updates I0708 11:42:48.540501 351518720 replica.cpp:537] Replica received write request for position 4 from (3799)@127.0.0.1:50325 I0708 11:42:48.540583 352055296 master.cpp:5835] Sending 1 offers to framework e055d60c-05ff-487e-82da-d0a43e52605c-0000 (default) I0708 11:42:48.540798 351518720 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 247us I0708 11:42:48.540868 351518720 replica.cpp:712] Persisted action at 4 I0708 11:42:48.540895 349908992 slave.cpp:1229] Forwarding total oversubscribed resources I0708 11:42:48.541035 352055296 master.cpp:5128] Received update of agent e055d60c-05ff-487e-82da-d0a43e52605c-S0 at slave(119)@127.0.0.1:50325 (localhost) with total oversubscribed resources I0708 11:42:48.541291 349908992 hierarchical.cpp:542] Agent e055d60c-05ff-487e-82da-d0a43e52605c-S0 (localhost) updated with oversubscribed resources (total: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000], allocated: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000]) I0708 11:42:48.541630 350982144 replica.cpp:691] Replica received learned notice for position 4 from @0.0.0.0:0 I0708 11:42:48.541911 350982144 leveldb.cpp:341] Persisting action (18 bytes) to leveldb took 189us I0708 11:42:48.541965 350982144 leveldb.cpp:399] Deleting ~2 keys from leveldb took 28us I0708 11:42:48.541987 350982144 replica.cpp:712] Persisted action at 4 I0708 11:42:48.542006 350982144 replica.cpp:697] Replica learned TRUNCATE action at position 4 I0708 11:42:48.544836 352055296 http.cpp:381] HTTP POST for /master/api/v1 from 127.0.0.1:50920 I0708 11:42:48.544884 352055296 http.cpp:484] Processing call SUBSCRIBE I0708 11:42:48.545382 352055296 master.cpp:7599] Added subscriber: a85e7341-ac15-4f18-9021-1a2efa326442 to the list of active subscribers I0708 11:42:48.550048 348835840 http.cpp:381] HTTP POST for /master/api/v1/scheduler from 127.0.0.1:50919 I0708 11:42:48.550339 348835840 master.cpp:3468] Processing ACCEPT call for offers: [ e055d60c-05ff-487e-82da-d0a43e52605c-O0 ] on agent e055d60c-05ff-487e-82da-d0a43e52605c-S0 at slave(119)@127.0.0.1:50325 (localhost) for framework e055d60c-05ff-487e-82da-d0a43e52605c-0000 (default) I0708 11:42:48.550390 348835840 mas...
Story points: 3