Repository: redis/redis

Issue:
[BUG]Sentinel  stuck with +sdown upon replica pod restart  
Hellow Team,  @yossigo , thanks for releasing the 6.2.0 with sentinel host name support and fixing all the related issues.

https://github.com/redis/redis/issues/8507
https://github.com/redis/redis/pull/8517
https://github.com/redis/redis/pull/8481
https://github.com/redis/redis/issues/8300 

We are evaluating to run sentinel based redis cluster on kubernetes, so  went ahead and verified [6.2.0](https://hub.docker.com/_/redis/) from Docker Hub , but surprisingly find out that, upon replica pod restart, `sentinel replicas mymaster` stuck with `    9) "flags"
   10) "s_down,slave"` state and never recovered.

```
127.0.0.1:26379> sentinel replicas mymaster
1)  1) "name"
    2) "redis-1.redis.FQDN"
    3) "ip"
    4) "redis-1.redis.FQDN"
    5) "port"
    6) "6379"
    7) "runid"
    8) "7b69bdd9dec849aa17bd9912b7e994cb6095b524"
    9) "flags"
   10) "s_down,slave"
   11) "link-pending-commands"
   12) "39"
   13) "link-refcount"
   14) "1"
   15) "last-ping-sent"
   16) "704052"
   17) "last-ok-ping-reply"
   18) "704597"
   19) "last-ping-reply"
   20) "704597"
   21) "s-down-time"
   22) "699026"
   23) "down-after-milliseconds"
   24) "5000"
   25) "info-refresh"
   26) "706516"
   27) "role-reported"
   28) "slave"
   29) "role-reported-time"
   30) "887409"
   31) "master-link-down-time"
   32) "0"
   33) "master-link-status"
   34) "ok"
   35) "master-host"
   36) "redis-0.redis.FQDN"
   37) "master-port"
   38) "6379"
   39) "slave-priority"
   40) "100"
   41) "slave-repl-offset"
   42) "66025"
```

last log on sentinel 

```
1:X 23 Feb 2021 21:21:47.545 # +sdown slave redis-1.redis.FQDN:6379 redis-1.redis.FQDN 6379 @ mymaster redis-0.redis.FQDN 6379
```


happy to provide more details if required, thanks!
[BUG] Failover not happening after enabling announce-hostnames & resolve-hostnames in sentinel
**Failover not happening once master gets restarted**

After enabling announce-hostnames & resolve-hostnames at first replica formation is happening without any issues.
But after master failover sentinels are not starting failover and I don't see ```+odown``` in sentinel logs which I see when running without hostnames.

**To reproduce**
Redis 6.2.4 on 3 pods along with sentinels on each with the below configurations.

**master.conf**
```
replica-announce-ip  <DNS>
```

**slave.conf**
```
replica-announce-ip  <SLAVE DNS>
replicaof <MASTER-DNS> <MASTER-PORT>
 ```
**sentinel.conf**
```
 SENTINEL resolve-hostnames yes
 SENTINEL announce-hostnames yes
 sentinel parallel-syncs mymaster 1
 replica-announce-ip  <Sentinel DNS>
```
Delete master pod and force it to not start again, see failover behavior and delete new master, and observe the same.

**Expected behavior**
Replica should become master when master went down.

**Additional information**
Below are the entries in logs when hostnames enabled where i only see ```+sdown```.

```
18:15:42.038 # +sdown master mymaster redis-0.redis.dev-wb.svc.cluster.local 6379
18:15:42.038 # +sdown sentinel eef584c8262dd5eaa3d8850e6932bfde80839a24 172.1.0.171 26379 @ mymaster redis-0.redis.dev-wb.svc.cluster.local 6379

```

Below is the log when hostnames were set to false, where I can see ```+odown```.

```
+sdown sentinel 9f8616a7f32bac48b49be67c2fce7039c0d16916 172.31.16.74 26379 @ RedisMaster 172.31.16.74 6379
+sdown master RedisMaster 172.31.16.74 6379
+odown master RedisMaster 172.31.16.74 6379 #quorum 2/2
+new-epoch 3
```



Discussion:
@satheeshaGowda It seems like sentinel still considers the connection up. This can happen if your instance changed an IP and the underlying network doesn't respond with ICMP or RST to packets of an old connection. Can you confirm this is the case?

Not very familiar with the Sentinel code base - I'd expect a reconnect timeout or at least a way to configure TCP keepalive on the sockets but I see none of that. @hwware maybe you're aware of something?
Hi @yossigo, thanks for taking a look and would like to confirm that indeed IP got changed but not sure about underneath network behavior, if required we can take tcp dump and analyze. 
@yossigo from my observation the link was down in 704 s ago, please see the following fields:
```
15) "last-ping-sent"
   16) "704052"
   17) "last-ok-ping-reply"
   18) "704597"
   19) "last-ping-reply"
   20) "704597"
```
@satheeshaGowda I think there is something wrong with your k8 network settings, can you confirm this is the case? Also it would be great if you can tell us your topology of your deployments for master, slaves and sentinels, thanks!
@satheeshaGowda , sorry just saw your message, thanks for confirming...
Here is our topology and relevant config.

<img width="692" alt="Screen Shot 2021-02-25 at 7 40 34 AM" src="https://user-images.githubusercontent.com/7914152/109177520-c1513580-773c-11eb-9184-9d309fb1473a.png">


Redis Kube Spec

```
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis
spec:
  serviceName: redis
  replicas: 3
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      initContainers:
      - name: config
        image: redis:6.2.0
        command: [ "bash", "-c" ]
        args:
          - |
            cp /tmp/redis/redis.conf /etc/redis/redis.conf

            MASTER_FQDN=redis-0.redis.REDACTED
            POD_FQDN=$(hostname -f)
            echo "replica-announce-ip  $POD_FQDN" >> /etc/redis/redis.conf
            echo "replica-announce-port 6379" >> /etc/redis/redis.conf
            if [ "$POD_FQDN" =  "$MASTER_FQDN" ]; then
                echo "this is master, not updating config..."
            else
                echo "updating replica redis.conf..."
                echo "replicaof $MASTER_FQDN 6379" >> /etc/redis/redis.conf
            fi
            cat /etc/redis/redis.conf
        volumeMounts:
        - name: redis-config
          mountPath: /etc/redis/
        - name: config
          mountPath: /tmp/redis/
      containers:
      - name: redis
        image: redis:6.2.0
        command: ["redis-server"]
        args: ["/etc/redis/redis.conf"]
        ports:
        - containerPort: 6379
          name: redis
        volumeMounts:
        - name: data
          mountPath: /data
        - name: redis-config
          mountPath: /etc/redis/
      volumes:
      - name: redis-config
        emptyDir: {}
      - name: config
        configMap:
          name: redis-config
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: scaleio
      resources:
        requests:
          storage: 100Mi

---
apiVersion: v1
kind: Service
metadata:
  name: redis
spec:
  clusterIP: None
  ports:
  - port: 6379
    targetPort: 6379
    name: redis
  selector:
    app: redis
```

Sentinel Kube Spec
```

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: sentinel
spec:
  serviceName: sentinel
  replicas: 3
  selector:
    matchLabels:
      app: sentinel
  template:
    metadata:
      labels:
        app: sentinel
    spec:
      initContainers:
      - name: config
        image: redis:6.2.0
        command: [ "sh", "-c" ]
        args:
          - |
            REDIS_PASSWORD=testpassword
            MASTER_FQDN=redis-0.redis.REDACTED
            POD_FQDN=$(hostname -f)
            echo "port 26379
            protected-mode no
            sentinel resolve-hostnames yes
            sentinel announce-hostnames yes
            sentinel announce-ip $POD_FQDN
            sentinel announce-port 26379
            sentinel monitor mymaster $MASTER_FQDN 6379 2
            sentinel down-after-milliseconds mymaster 5000
            sentinel failover-timeout mymaster 60000
            sentinel parallel-syncs mymaster 1
            sentinel auth-pass mymaster $REDIS_PASSWORD
            " > /etc/redis/sentinel.conf
            cat /etc/redis/sentinel.conf
        volumeMounts:
        - name: redis-config
          mountPath: /etc/redis/
      containers:
      - name: sentinel
        image: redis:6.2.0
        command: ["redis-server", "/etc/redis/sentinel.conf", "--sentinel"]
        ports:
        - containerPort: 26379
          name: sentinel
        volumeMounts:
        - name: redis-config
          mountPath: /etc/redis/
        - name: data
          mountPath: /data
      volumes:
      - name: redis-config
        emptyDir: {}
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: scaleio
      resources:
        requests:
          storage: 100Mi

---
apiVersion: v1
kind: Service
metadata:
  name: sentinel
spec:
  clusterIP: None
  ports:
  - port: 26379
    targetPort: 26379
    name: sentinel
  selector:
    app: sentinel
```

@satheeshaGowda I did not find something specifically goes wrong in your config, maybe is there any change happens during runtime?
No, not really, and its always reproducible @hwware 
@satheeshaGowda I think the best way to confirm this is to use `netstat` on the sentinel node and see if there are still TCP connections established with the Redis at the old address. Assuming that's the case, we'll have to address that in one of two ways:

1) Make Sentinel explicitly reconnect after some inactivity time.
2) Use TCP_KEEPALIVE to make the socket drop itself after some time.

[1] will probably be a better fix in the long run, but [2] is easier.

This is a fairly fundamental issue, I guess this never came up before because any situation where the instance's IP changed could not be supported anyway.
Having this issue on Kubernetes with AWS CNI plugin. Can confirm that old IP address still appears in netstat output.

The same applies to replicas. If I delete a replica pod, Sentinel does not pick up the replica with the new IP. Connection to the old one is still present with SYN_SENT status.
Having this issue on Kubernetes 1.20
![image](https://user-images.githubusercontent.com/35965787/114362159-9c5c3900-9ba9-11eb-8609-586a7611ffe2.png)
I found that when K8s node0 is down，the hostname ‘redis-proton-redis-0.redis-proton-redis.redis.svc.cluster.local’ can not be resolve to ip address
in this time, sentinel 'redis-1' will send request 'is-master-down-by-addr' to sentinel 'redis-2',
sentinel 'redis-2' will call function ‘getSentinelRedisInstanceByAddrAndRunID’ want to find the correct instance，but failed。
![image](https://user-images.githubusercontent.com/35965787/114363636-2ce74900-9bab-11eb-9ae6-bf730c0359c1.png)
![image](https://user-images.githubusercontent.com/35965787/114363646-32dd2a00-9bab-11eb-9b3b-899be778833f.png)


@juris A `SYN_SENT` entry in netstat implies it's attempting to reconnect, I was actually expecting to find an `ESTABLISHED` entry which would indicate the connection did not drop and hangs in there due to lack of timeout.

@kenlee1988 I'm not sure your report is related to the previous issues but there's certainly a real issue there. I've put together a quick patch, but unfortunately I don't have the bandwidth at the moment to set up an environment and test it properly or validate that all similar cases are indeed covered. I'll PR that and if anyone can pick this up and test/improve that can be a great help.
Might be the same issue as https://github.com/redis/redis/issues/9103?
@sivanagireddyb 
Hi I try to reproduce your problems in the 6.2.4 version, but even I set sentinel resolve-hostnames is yes and  sentinel announce-hostnames is yes, I could get the "odown" information in the logs and Replica should become master.   Following is my test profile, test steps and output logs,  Let me know you have any questions, thanks

**_Part 1.  Test Profile -- I have 1 master redis instance, 2 repli instances and 3 sentinels_**

**_master.conf:_**

replica-announce-ip testHost
port 6380
protected-mode no

**_replica1.conf_**

replica-announce-ip testHost
port 6381
protected-mode no
replicaof testHost 6380

**_replica2.conf_**

replica-announce-ip testHost
port 6382
protected-mode no
replicaof testHost 6380

**_sentinel1.conf:_**

sentinel monitor mymaster testHost 6380 2
sentinel down-after-milliseconds mymaster 6000
sentinel failover-timeout mymaster 18000
replica-announce-ip testHost
port 26380
sentinel resolve-hostnames yes
sentinel announce-hostnames yes

**_sentinel2.conf:_**

sentinel monitor mymaster testHost 6380 2
sentinel down-after-milliseconds mymaster 6000
sentinel failover-timeout mymaster 18000
replica-announce-ip testHost
port 26381
sentinel resolve-hostnames yes
sentinel announce-hostnames yes

**_sentinel3.conf:_**

sentinel monitor mymaster testHost 6380 2
sentinel down-after-milliseconds mymaster 6000
sentinel failover-timeout mymaster 18000
replica-announce-ip testHost
port 26382
sentinel resolve-hostnames yes
sentinel announce-hostnames yes

**_Part 2.  Test Step_**

**_after all master, replicas, and sentinels startup, first shutdown sentinel1, and wait for a few seconds,
  shutdown master, and never startup again_**

**_Part 3 Output logs:_**

**_Sentinel 2 log:_**

**_3271:X 14 Jul 2021 12:02:33.033 # +sdown master mymaster testHost 6380_**
**_3271:X 14 Jul 2021 12:02:33.107 # +odown master mymaster testHost 6380 #quorum 2/2_**
3271:X 14 Jul 2021 12:02:33.107 # +new-epoch 1
3271:X 14 Jul 2021 12:02:33.107 # +try-failover master mymaster testHost 6380
3271:X 14 Jul 2021 12:02:33.116 # +vote-for-leader d82a59bbe49eb5b98607f9c3c56eaab8d137d1c8 1
3271:X 14 Jul 2021 12:02:33.131 # ee04e963aa001bf423055f20af65ad3630d8c3e7 voted for d82a59bbe49eb5b98607f9c3c56eaab8d137d1c8 1
3271:X 14 Jul 2021 12:02:33.188 # +elected-leader master mymaster testHost 6380
3271:X 14 Jul 2021 12:02:33.188 # +failover-state-select-slave master mymaster testHost 6380
3271:X 14 Jul 2021 12:02:33.264 # +selected-slave slave testHost:6381 testHost 6381 @ mymaster testHost 6380
3271:X 14 Jul 2021 12:02:33.264 * +failover-state-send-slaveof-noone slave testHost:6381 testHost 6381 @ mymaster testHost 6380
3271:X 14 Jul 2021 12:02:33.332 * +failover-state-wait-promotion slave testHost:6381 testHost 6381 @ mymaster testHost 6380
3271:X 14 Jul 2021 12:02:34.188 # +promoted-slave slave testHost:6381 testHost 6381 @ mymaster testHost 6380
3271:X 14 Jul 2021 12:02:34.189 # +failover-state-reconf-slaves master mymaster testHost 6380
3271:X 14 Jul 2021 12:02:34.243 * +slave-reconf-sent slave testHost:6382 testHost 6382 @ mymaster testHost 6380
3271:X 14 Jul 2021 12:02:35.196 * +slave-reconf-inprog slave testHost:6382 testHost 6382 @ mymaster testHost 6380
3271:X 14 Jul 2021 12:02:35.196 * +slave-reconf-done slave testHost:6382 testHost 6382 @ mymaster testHost 6380
3271:X 14 Jul 2021 12:02:35.283 # -odown master mymaster testHost 6380
3271:X 14 Jul 2021 12:02:35.283 # +failover-end master mymaster testHost 6380
**_3271:X 14 Jul 2021 12:02:35.283 # +switch-master mymaster testHost 6380 testHost 6381_**
3271:X 14 Jul 2021 12:02:35.283 * +slave slave testHost:6382 testHost 6382 @ mymaster testHost 6381
3271:X 14 Jul 2021 12:02:35.283 * +slave slave testHost:6380 testHost 6380 @ mymaster testHost 6381
3271:X 14 Jul 2021 12:02:41.315 # +sdown slave testHost:6380 testHost 6380 @ mymaster testHost 6381

**_Sentinel 3 log:_**

**_3276:X 14 Jul 2021 12:02:32.938 # +sdown master mymaster testHost 6380_**
3276:X 14 Jul 2021 12:02:33.124 # +new-epoch 1
3276:X 14 Jul 2021 12:02:33.131 # +vote-for-leader d82a59bbe49eb5b98607f9c3c56eaab8d137d1c8 1
**_3276:X 14 Jul 2021 12:02:34.075 # +odown master mymaster testHost 6380 #quorum 2/2_**
3276:X 14 Jul 2021 12:02:34.075 # Next failover delay: I will not start a failover before Wed Jul 14 12:03:09 2021
3276:X 14 Jul 2021 12:02:34.243 # +config-update-from sentinel d82a59bbe49eb5b98607f9c3c56eaab8d137d1c8 127.0.0.1 26381 @ mymaster testHost 6380
**_3276:X 14 Jul 2021 12:02:34.243 # +switch-master mymaster testHost 6380 testHost 6381_**
3276:X 14 Jul 2021 12:02:34.243 * +slave slave testHost:6382 testHost 6382 @ mymaster testHost 6381
3276:X 14 Jul 2021 12:02:34.243 * +slave slave testHost:6380 testHost 6380 @ mymaster testHost 6381
3276:X 14 Jul 2021 12:02:40.272 # +sdown slave testHost:6380 testHost 6380 @ mymaster testHost 6381

From the log, we can see The replica testHost:6381 is promoted to master.

@hwware use DNS/hostnames names instead of ips
> @hwware use DNS/hostnames names instead of ips

Hi @SivaBu-kore  I update the config file, replace the ips with hostname, after shutdown one of the sentinels and master, replica still could be promoted to master.  Please verify my reproduced steps and let me know any concerns. 

Thanks
@hmware I have tested with 1 Master & 1 Replica. You are testing with 1 Master & 2 Replicas. I'm not sure it will be an issue. 
@SivaBu-kore Following is my test result with 1 Master & 1 Replica,  it looks Replica still could be promoted to Master. 
@sivanagireddyb Please check the following log with 1 Master & 1 Replica

Please let me know if some test steps are not consistent with you, thanks

**_Part 1. Test Profile -- I have 1 master redis instance, 1 repli instances and 3 sentinels_**

**_master.conf:_**

replica-announce-ip testHost
port 6380
protected-mode no

**_replica1.conf_**

replica-announce-ip testHost
port 6381
protected-mode no
replicaof testHost 6380

**_sentinel1.conf:_**

sentinel monitor mymaster testHost 6380 2
sentinel down-after-milliseconds mymaster 6000
sentinel failover-timeout mymaster 18000
replica-announce-ip testHost
port 26380
sentinel resolve-hostnames yes
sentinel announce-hostnames yes

**_sentinel2.conf:_**

sentinel monitor mymaster testHost 6380 2
sentinel down-after-milliseconds mymaster 6000
sentinel failover-timeout mymaster 18000
replica-announce-ip testHost
port 26381
sentinel resolve-hostnames yes
sentinel announce-hostnames yes

**_sentinel3.conf:_**

sentinel monitor mymaster testHost 6380 2
sentinel down-after-milliseconds mymaster 6000
sentinel failover-timeout mymaster 18000
replica-announce-ip testHost
port 26382
sentinel resolve-hostnames yes
sentinel announce-hostnames yes

**Part 2. Test Step**

after all master, replicas, and sentinels startup, **_first shutdown sentinel1, and wait for a few seconds,
shutdown master_**, and never startup again

**Part 3 Output logs:**

**_Sentinel 2 log:_**

**_2495:X 15 Jul 2021 09:45:01.584 # +sdown master mymaster testHost 6380_**
2495:X 15 Jul 2021 09:45:01.695 # +new-epoch 1
2495:X 15 Jul 2021 09:45:01.701 # +vote-for-leader 72c18be5a45f6a5fed5bc9d1e1ab6067510ad4a4 1
**_2495:X 15 Jul 2021 09:45:02.724 # +odown master mymaster testHost 6380 #quorum 2/2_**
2495:X 15 Jul 2021 09:45:02.724 # Next failover delay: I will not start a failover before Thu Jul 15 09:45:38 2021
2495:X 15 Jul 2021 09:45:02.952 # +config-update-from sentinel 72c18be5a45f6a5fed5bc9d1e1ab6067510ad4a4 127.0.0.1 26382 @ mymaster testHost 6380
2495:X 15 Jul 2021 09:45:02.952 # +switch-master mymaster testHost 6380 testHost 6381
2495:X 15 Jul 2021 09:45:02.953 * +slave slave testHost:6380 testHost 6380 @ mymaster testHost 6381
2495:X 15 Jul 2021 09:45:09.053 # +sdown slave testHost:6380 testHost 6380 @ mymaster testHost 6381

**_Sentinel 3 log:_**

**_2500:X 15 Jul 2021 09:45:01.588 # +sdown master mymaster testHost 6380_**
**_2500:X 15 Jul 2021 09:45:01.660 # +odown master mymaster testHost 6380 #quorum 2/2_**
2500:X 15 Jul 2021 09:45:01.660 # +new-epoch 1
2500:X 15 Jul 2021 09:45:01.660 # +try-failover master mymaster testHost 6380
2500:X 15 Jul 2021 09:45:01.687 # +vote-for-leader 72c18be5a45f6a5fed5bc9d1e1ab6067510ad4a4 1
2500:X 15 Jul 2021 09:45:01.701 # c6a6398e2873710278c4a5ea5ed14aea0fe98465 voted for 72c18be5a45f6a5fed5bc9d1e1ab6067510ad4a4 1
2500:X 15 Jul 2021 09:45:01.744 # +elected-leader master mymaster testHost 6380
2500:X 15 Jul 2021 09:45:01.744 # +failover-state-select-slave master mymaster testHost 6380
2500:X 15 Jul 2021 09:45:01.841 # +selected-slave slave testHost:6381 testHost 6381 @ mymaster testHost 6380
2500:X 15 Jul 2021 09:45:01.841 * +failover-state-send-slaveof-noone slave testHost:6381 testHost 6381 @ mymaster testHost 6380
2500:X 15 Jul 2021 09:45:01.908 * +failover-state-wait-promotion slave testHost:6381 testHost 6381 @ mymaster testHost 6380
2500:X 15 Jul 2021 09:45:02.883 # +promoted-slave slave testHost:6381 testHost 6381 @ mymaster testHost 6380
2500:X 15 Jul 2021 09:45:02.883 # +failover-state-reconf-slaves master mymaster testHost 6380
2500:X 15 Jul 2021 09:45:02.945 # +failover-end master mymaster testHost 6380
2500:X 15 Jul 2021 09:45:02.945 # +switch-master mymaster testHost 6380 testHost 6381
2500:X 15 Jul 2021 09:45:02.945 * +slave slave testHost:6380 testHost 6380 @ mymaster testHost 6381
2500:X 15 Jul 2021 09:45:08.979 # +sdown slave testHost:6380 testHost 6380 @ mymaster testHost 6381


@hwware i am also running into this issue attempting to use sentinel in a docker swarm environment (where the internal DNS can resolve task names to addresses because ip's may change. running on an overlay network - where nat should not be an issue as all ports are open and mapped 1:1)

below is the simplest compose file which is basically a copy of your suggested configuration, and the issue reproduces both on a single machine, as well as on a cluster.

```
version: '3.8'

services:
  master:
    image: redis:alpine
    deploy:
      replicas: 1
      restart_policy:
        delay: 5s
    networks:
      - redis-net
    hostname: master
    volumes:
      - type: volume
        source: master-data
        target: /data
    entrypoint:
      - "sh"
      - "-c"
      - |
        cat <<EOF > /data/redis.conf
        replica-announce-ip tasks.master
        protected-mode no
        bind 0.0.0.0
        EOF
        docker-entrypoint.sh redis-server /data/redis.conf --loglevel debug

  replica:
    image: redis:alpine
    deploy:
      replicas: 1
      restart_policy:
        delay: 5s
    networks:
      - redis-net
    hostname: replica
    volumes:
      - type: volume
        source: replica-data
        target: /data
    environment:
      - REPLICA_PRIORITY={{.Task.Slot}}
      - REPLICA_ADDR={{.Task.Name}}
    entrypoint:
      - "sh"
      - "-c"
      - |
        cat <<EOF >> /data/redis.conf
        protected-mode no
        bind 0.0.0.0
        replica-announce-ip $$REPLICA_ADDR
        replicaof tasks.master 6379
        replica-priority $$REPLICA_PRIORITY
        EOF
        docker-entrypoint.sh redis-server /data/redis.conf --loglevel debug

  sentinel:
    image: redis:alpine
    deploy:
      replicas: 3
      restart_policy:
        delay: 5s
    networks:
      - redis-net
    hostname: sentinel-{{.Task.Slot}}
    environment:
      - SENTINEL_ADDR={{.Task.Name}}
    entrypoint:
      - "sh"
      - "-c"
      - |
        cat <<EOF >> /data/sentinel.conf
        protected-mode no
        bind 0.0.0.0
        sentinel announce-ip $$SENTINEL_ADDR
        sentinel monitor test-cluster tasks.master 6379 2
        sentinel down-after-milliseconds test-cluster 6000
        sentinel failover-timeout test-cluster 18000
        sentinel resolve-hostnames yes
        sentinel announce-hostnames yes
        EOF
        docker-entrypoint.sh redis-sentinel /data/sentinel.conf --loglevel debug

networks:
  redis-net:
    driver: overlay
    attachable: true

volumes:
  master-data:
    name: 'redis_volume_{{.Service.Name}}_{{.Task.Slot}}'
  replica-data:
    name: 'redis_volume_{{.Service.Name}}_{{.Task.Slot}}'
```

having sentinel announce-hostnames yes results in sentinels emitting the +sdown message for the master but never moving on to +odown and failover.
simply changing sentinel announce-hostnames to no in the compose file fixes this without any other change. 
this is a problem because it would need to pin the master's ip address and in case of a machine failure the master would never be re-added to the set (there is no record in the sentinel's config/history of the original master), not even as a replica.

you can run the compose file using `docker stack deploy -c sentinel-compose.yml test` with docker running in swarm mode
I'm having same issue and I think I was able to localise it.

---
Here is my investigation of I got when I running `alpine` based image on k8s as a StatefulSet

Check IP on `redis-0.redis-headless.default.svc.cluster.local` pod:
```bash
/data # ifconfig
eth0      Link encap:Ethernet 
              inet addr:10.244.0.161
```

Starting `redis-sentinel` pod and we can see it is connecting to `redis-0` right away:

```bash
/data # redis-cli
127.0.0.1:6379> monitor
OK
1629506678.413582 [0 10.244.0.177:53103] "PING"
1629506678.413711 [0 10.244.0.177:53103] "INFO"
1629506678.413838 [0 10.244.0.177:42955] "SUBSCRIBE" "__sentinel__:hello"
1629506679.513769 [0 10.244.0.177:53103] "PING"
1629506680.463244 [0 10.244.0.177:53103] "PUBLISH" "__sentinel__:hello" "10.244.0.177,26379,e51cc7822a48f34ae85f8cf0faa8f0a75ff6a243,0,mymaster,redis-0.redis-headless.default.svc.cluster.local,6379,0"
1629506680.534972 [0 10.244.0.177:53103] "PING"
1629506681.536829 [0 10.244.0.177:53103] "PING"
1629506682.512930 [0 10.244.0.177:53103] "PUBLISH" "__sentinel__:hello" "10.244.0.177,26379,e51cc7822a48f34ae85f8cf0faa8f0a75ff6a243,0,mymaster,redis-0.redis-headless.default.svc.cluster.local,6379,0"
1629506682.579558 [0 10.244.0.177:53103] "PING"
```

Checking networking info for our `redis-sentinel-0` pod:
```bash
❯ kubectl exec -it redis-sentinel-0 -- /bin/sh
Defaulted container "redis-sentinel" out of: redis-sentinel, config (init)
/data # netstat
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 redis-sentinel-0.redis-sentinel.default.svc.cluster.local:42955 redis-0.redis-headless.default.svc.cluster.local:redis ESTABLISHED
tcp        0      0 redis-sentinel-0.redis-sentinel.default.svc.cluster.local:34497 redis-1.redis-headless.default.svc.cluster.local:redis ESTABLISHED
tcp        0      0 redis-sentinel-0.redis-sentinel.default.svc.cluster.local:59095 redis-1.redis-headless.default.svc.cluster.local:redis TIME_WAIT
tcp        0      0 redis-sentinel-0.redis-sentinel.default.svc.cluster.local:53103 redis-0.redis-headless.default.svc.cluster.local:redis ESTABLISHED
tcp        0      0 redis-sentinel-0.redis-sentinel.default.svc.cluster.local:39105 redis-1.redis-headless.default.svc.cluster.local:redis ESTABLISHED
tcp        0      0 redis-sentinel-0.redis-sentinel.default.svc.cluster.local:41515 redis-1.redis-headless.default.svc.cluster.local:redis TIME_WAIT
Active UNIX domain sockets (w/o servers)
Proto RefCnt Flags       Type       State         I-Node Path
```

Now we are killing `redis-0` pod with `kubectl delete pod redis-0` and checking it's new IP address:
```bash
/data # ifconfig
eth0      Link encap:Ethernet 
              inet addr:10.244.0.237
```

Now let's check networking info for our `redis-sentinel-0` pod one more time:

```bash
❯ kubectl exec -it redis-sentinel-0 -- /bin/sh
Defaulted container "redis-sentinel" out of: redis-sentinel, config (init)
/data # netstat
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 redis-sentinel-0.redis-sentinel.default.svc.cluster.local:34497 redis-1.redis-headless.default.svc.cluster.local:redis ESTABLISHED
tcp        0      0 redis-sentinel-0.redis-sentinel.default.svc.cluster.local:59095 redis-1.redis-headless.default.svc.cluster.local:redis TIME_WAIT
tcp        0      0 redis-sentinel-0.redis-sentinel.default.svc.cluster.local:39105 redis-1.redis-headless.default.svc.cluster.local:redis ESTABLISHED
tcp        0      0 redis-sentinel-0.redis-sentinel.default.svc.cluster.local:41515 redis-1.redis-headless.default.svc.cluster.local:redis TIME_WAIT
tcp        0      1 redis-sentinel-0.redis-sentinel.default.svc.cluster.local:55851 10.244.0.161:redis      SYN_SENT
tcp        0      1 redis-sentinel-0.redis-sentinel.default.svc.cluster.local:52127 10.244.0.161:redis      SYN_SENT
Active UNIX domain sockets (w/o servers)
Proto RefCnt Flags       Type       State         I-Node Path
```

Note those hanging `SYN_SENT` that still using `10.244.0.161` (which was initial IP of `redis-0`) 
```bash
tcp        0      1 redis-sentinel-0.redis-sentinel.default.svc.cluster.local:55851 10.244.0.161:redis      SYN_SENT
tcp        0      1 redis-sentinel-0.redis-sentinel.default.svc.cluster.local:52127 10.244.0.161:redis      SYN_SENT
```

At the end after killing pods several times it looks like this:
```bash
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      1 10.244.0.211:40581      10.244.0.201:6379       SYN_SENT    1/redis-sentinel 0.
tcp        0      1 10.244.0.211:41353      10.244.1.38:6379        SYN_SENT    1/redis-sentinel 0.
tcp        0      1 10.244.0.211:44925      10.244.0.201:6379       SYN_SENT    1/redis-sentinel 0.
tcp        0      0 10.244.0.211:26379      10.244.0.234:46771      ESTABLISHED 1/redis-sentinel 0.
tcp        0      0 10.244.0.211:26379      10.244.1.10:33485       ESTABLISHED 1/redis-sentinel 0.
tcp        0      0 10.244.0.211:33909      10.244.1.10:26379       ESTABLISHED 1/redis-sentinel 0.
tcp        0      0 10.244.0.211:54913      10.244.0.234:26379      ESTABLISHED 1/redis-sentinel 0.
tcp        0      1 10.244.0.211:36375      10.244.1.38:6379        SYN_SENT    1/redis-sentinel 0.
```
It looks like Sentinel is memorising IP after hostname lookup and keep trying to connect to that address.

Here `10.244.1.32` is IP address that used to be valid for `redis-1` before that pod was restarted (after restart DNS name for a StatefulSet pod is the same, but IP address is a not preserved) 

```
root@redis-sentinel-0:/data# netstat -etd
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode
tcp        0      0 redis-sentinel-0.:48011 redis-0.redis-head:6379 ESTABLISHED root       64189289
tcp        0      0 redis-sentinel-0.:52896 151.101.18.132:80       TIME_WAIT   root       0
tcp        0      0 redis-sentinel-0.:43725 10-244-1-81.redis:26379 ESTABLISHED root       64185624
tcp        0      0 redis-sentinel-0.:47791 redis-0.redis-head:6379 ESTABLISHED root       64189288
tcp        0      1 redis-sentinel-0.:44463 10.244.1.32:6379        SYN_SENT    root       66661160
tcp        0      0 redis-sentinel-0.:54909 10-244-0-194.redi:26379 ESTABLISHED root       64184651
tcp        0      0 redis-sentinel-0.:26379 10-244-0-194.redi:55715 ESTABLISHED root       64184635
tcp        0      0 redis-sentinel-0.:26379 10-244-1-81.redis:54561 ESTABLISHED root       64184707
tcp        0      1 redis-sentinel-0.:36145 10.244.1.32:6379        SYN_SENT    root       66661984
root@redis-sentinel-0:/data# tcpdump host 10.244.1.32
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
23:24:20.482923 IP redis-sentinel-0.redis-sentinel.default.svc.cluster.local.36937 > 10.244.1.32.6379: Flags [S], seq 689568883, win 64240, options [mss 1460,sackOK,TS val 567563557 ecr 0,nop,wscale 7], length 0
23:24:25.165632 IP redis-sentinel-0.redis-sentinel.default.svc.cluster.local.53155 > 10.244.1.32.6379: Flags [S], seq 135490147, win 64240, options [mss 1460,sackOK,TS val 567568239 ecr 0,nop,wscale 7], length 0
23:24:26.178933 IP redis-sentinel-0.redis-sentinel.default.svc.cluster.local.53155 > 10.244.1.32.6379: Flags [S], seq 135490147, win 64240, options [mss 1460,sackOK,TS val 567569253 ecr 0,nop,wscale 7], length 0
23:24:28.194907 IP redis-sentinel-0.redis-sentinel.default.svc.cluster.local.53155 > 10.244.1.32.6379: Flags [S], seq 135490147, win 64240, options [mss 1460,sackOK,TS val 567571269 ecr 0,nop,wscale 7], length 0
23:24:28.433471 IP redis-sentinel-0.redis-sentinel.default.svc.cluster.local.43915 > 10.244.1.32.6379: Flags [S], seq 4200356803, win 64240, options [mss 1460,sackOK,TS val 567571507 ecr 0,nop,wscale 7], length 0
23:24:29.442940 IP redis-sentinel-0.redis-sentinel.default.svc.cluster.local.43915 > 10.244.1.32.6379: Flags [S], seq 4200356803, win 64240, options [mss 1460,sackOK,TS val 567572517 ecr 0,nop,wscale 7], length 0
23:24:31.458932 IP redis-sentinel-0.redis-sentinel.default.svc.cluster.local.43915 > 10.244.1.32.6379: Flags [S], seq 4200356803, win 64240, options [mss 1460,sackOK,TS val 567574533 ecr 0,nop,wscale 7], length 0```
I'm not familiar with Redis codebase, but it seems like it indeed stores `ip:port` after doing hostname lookup
E.g. https://github.com/redis/redis/blob/63e2a6d212e9a30d9768b1d044348420e5b128c9/src/sentinel.c#L1427-L1437
Also running into this issue, any resolution?
I am also facing same issue . After using hostname in place of ip , failover is not happening . 
Can you please share any fix .
> 

@hwware why you have used same hostname 'testhost'  for all replica ?  
Indeed, even though Sentinel using hostnames, it doesn't support dynamic IPs unfortunately. 
I will evaluate the required effort to support it. 
Thanks.

Patch:
diff --git a/src/sentinel.c b/src/sentinel.c
index e309810395c..5bfb5ea03d3 100644
--- a/src/sentinel.c
+++ b/src/sentinel.c
@@ -552,10 +552,10 @@ void sentinelIsRunning(void) {
 
 /* Create a sentinelAddr object and return it on success.
  * On error NULL is returned and errno is set to:
- *  ENOENT: Can't resolve the hostname.
+ *  ENOENT: Can't resolve the hostname, unless accept_unresolved is non-zero.
  *  EINVAL: Invalid port number.
  */
-sentinelAddr *createSentinelAddr(char *hostname, int port) {
+sentinelAddr *createSentinelAddr(char *hostname, int port, int is_accept_unresolved) {
     char ip[NET_IP_STR_LEN];
     sentinelAddr *sa;
 
@@ -565,8 +565,14 @@ sentinelAddr *createSentinelAddr(char *hostname, int port) {
     }
     if (anetResolve(NULL,hostname,ip,sizeof(ip),
                     sentinel.resolve_hostnames ? ANET_NONE : ANET_IP_ONLY) == ANET_ERR) {
-        errno = ENOENT;
-        return NULL;
+        serverLog(LL_WARNING, "Failed to resolve hostname '%s'", hostname);
+        if (sentinel.resolve_hostnames && is_accept_unresolved) {
+            ip[0] = '\0';
+        }
+        else {
+            errno = ENOENT;
+            return NULL;
+        }
     }
     sa = zmalloc(sizeof(*sa));
     sa->hostname = sdsnew(hostname);
@@ -598,6 +604,15 @@ int sentinelAddrIsEqual(sentinelAddr *a, sentinelAddr *b) {
     return a->port == b->port && !strcasecmp(a->ip,b->ip);
 }
 
+/* Return non-zero if the two addresses are equal, either by address
+ * or by hostname if they could not have been resolved.
+ */
+int sentinelAddrOrHostnameEqual(sentinelAddr *a, sentinelAddr *b) {
+    return a->port == b->port &&
+            (!strcmp(a->ip, b->ip)  ||
+            !strcasecmp(a->hostname, b->hostname));
+}
+
 /* Return non-zero if a hostname matches an address. */
 int sentinelAddrEqualsHostname(sentinelAddr *a, char *hostname) {
     char ip[NET_IP_STR_LEN];
@@ -1259,7 +1274,7 @@ sentinelRedisInstance *createSentinelRedisInstance(char *name, int flags, char *
     serverAssert((flags & SRI_MASTER) || master != NULL);
 
     /* Check address validity. */
-    addr = createSentinelAddr(hostname,port);
+    addr = createSentinelAddr(hostname,port,1);
     if (addr == NULL) return NULL;
 
     /* For slaves use ip/host:port as name. */
@@ -1387,7 +1402,7 @@ sentinelRedisInstance *sentinelRedisInstanceLookupSlave(
      * If that is the case, depending on configuration we either resolve
      * it and use the IP address or fail.
      */
-    addr = createSentinelAddr(slave_addr, port);
+    addr = createSentinelAddr(slave_addr, port, 0);
     if (!addr) return NULL;
     key = announceSentinelAddrAndPort(addr);
     releaseSentinelAddr(addr);
@@ -1450,8 +1465,10 @@ sentinelRedisInstance *getSentinelRedisInstanceByAddrAndRunID(dict *instances, c
 
     serverAssert(addr || runid);   /* User must pass at least one search param. */
     if (addr != NULL) {
-        /* Resolve addr, we use the IP as a key even if a hostname is used */
-        ri_addr = createSentinelAddr(addr, port);
+        /* Try to resolve addr. If hostnames are used, we're accepting an ri_addr
+         * that contains an hostname only and can still be matched based on that.
+         */
+        ri_addr = createSentinelAddr(addr,port,1);
         if (!ri_addr) return NULL;
     }
     di = dictGetIterator(instances);
@@ -1460,8 +1477,7 @@ sentinelRedisInstance *getSentinelRedisInstanceByAddrAndRunID(dict *instances, c
 
         if (runid && !ri->runid) continue;
         if ((runid == NULL || strcmp(ri->runid, runid) == 0) &&
-            (addr == NULL || (strcmp(ri->addr->ip, ri_addr->ip) == 0 &&
-                            ri->addr->port == port)))
+            (addr == NULL || sentinelAddrOrHostnameEqual(ri->addr, ri_addr)))
         {
             instance = ri;
             break;
@@ -1565,7 +1581,7 @@ int sentinelResetMasterAndChangeAddress(sentinelRedisInstance *master, char *hos
     dictIterator *di;
     dictEntry *de;
 
-    newaddr = createSentinelAddr(hostname,port);
+    newaddr = createSentinelAddr(hostname,port,0);
     if (newaddr == NULL) return C_ERR;
 
     /* There can be only 0 or 1 slave that has the newaddr.
@@ -2353,6 +2369,7 @@ static int instanceLinkNegotiateTLS(redisAsyncContext *context) {
  * is disconnected. Note that link->disconnected is true even if just
  * one of the two links (commands and pub/sub) is missing. */
 void sentinelReconnectInstance(sentinelRedisInstance *ri) {
+
     if (ri->link->disconnected == 0) return;
     if (ri->addr->port == 0) return; /* port == 0 means invalid address. */
     instanceLink *link = ri->link;
@@ -2363,7 +2380,21 @@ void sentinelReconnectInstance(sentinelRedisInstance *ri) {
 
     /* Commands connection. */
     if (link->cc == NULL) {
+
+        /* It might be that the instance is disconnected because it wasn't available earlier when the instance
+         * allocated, say during failover, and therefore we failed to resolve its ip.
+         * Another scenario is that the instance restarted with new ip, and we should resolve its new ip based on
+         * its hostname */
+        if (sentinel.resolve_hostnames) {
+            sentinelAddr *tryResolveAddr = createSentinelAddr(ri->addr->hostname, ri->addr->port, 0);
+            if (tryResolveAddr != NULL) {
+                releaseSentinelAddr(ri->addr);
+                ri->addr = tryResolveAddr;
+            }
+        }
+
         link->cc = redisAsyncConnectBind(ri->addr->ip,ri->addr->port,server.bind_source_addr);
+
         if (link->cc && !link->cc->err) anetCloexec(link->cc->c.fd);
         if (!link->cc) {
             sentinelEvent(LL_DEBUG,"-cmd-link-reconnection",ri,"%@ #Failed to establish connection");
@@ -4886,6 +4917,7 @@ sentinelRedisInstance *sentinelSelectSlave(sentinelRedisInstance *master) {
     max_master_down_time += master->down_after_period * 10;
 
     di = dictGetIterator(master->slaves);
+
     while((de = dictNext(di)) != NULL) {
         sentinelRedisInstance *slave = dictGetVal(de);
         mstime_t info_validity_time;