Repository: goauthentik/authentik

Issue:
Proxy Provider not working 
Hello

Please help me, I updated the authentik server to 23.1.2, it worked perfectly until now, now the Proxy Provider is not working because of the following error

this is in the server log

{"error":"Post \"https://auth.xxx.com/application/o/token/\": dial tcp 192.168.10.240:443: connect: connection refused","event":"failed to redeem code","level":"warning","logger":"authentik.outpost.proxyv2.application","name":"Kuma","timestamp":"2023-01-24T13:01:34Z"}

The IP in the log is the IP of the nginx reverse proxy manager. The proxy works properly, I don't see any errors. Anyone have any ideas?



Discussion:
I am having the same issue with 2023.1.2 and 2023.1.1. Had to downgrade to 2023.1.0 to resolve the error.
```
{"error":"Post \"https://auth.xxxx.xxxx/application/o/token/\": dial tcp [fd00:1720:180:240::4]:443: connect: connection refused","event":"failed to redeem code","level":"warning","logger":"authentik.outpost.proxyv2.application","name":"xxxx","timestamp":"2023-01-24T08:25:44Z"}
```
I am running this setup in docker pretty much using the example `compose.yml` in the project where the ip is the internal docker network ip address of the `server` container.
yes, if I restore the container, it works perfectly again, but I would be more happy with an updated version :)
Starting with 2023.1, the outpost uses the external URL to get OAuth tokens, which is the same URL that is configured in the outpost settings
yes, that's right, it's set up like this, the proxy must go to https://aut.xxxx.com. this has happened so far, there was no problem with it, but unfortunately with the new update it now says that connection refused. 
there was no change in the network, just like the dns, only the update, I don't really understand the matter and I don't have much of an idea
Having the same issue. My authentik dashboard/config is accessible at authentik.[mydomain].com. I have a public dns entry for this page, and I have a wildcard split dns locally pointing to my server. 2023.1.1 and .2 have the same issue as reported here. Downgrading resolves the issue.

When I look at my outpost setting it makes reference to authentik.[mydomain].com. 
This sounds like a routing issue. Try resolving the host and connecting from the container.

Replace `example.com` and port 80 with your own service.

```python
# ak shell
### authentik shell (2023.1.2)
### Node 25e194f2a8ac | Arch x86_64 | Python 3.11.1
>>> import socket
>>> socket.gethostbyname('example.com')
'93.184.216.34'
>>> x = socket.getaddrinfo('example.com', 80, socket.AF_INET, proto=socket.IPPROTO_TCP)
>>> x
[(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('93.184.216.34', 80))]
>>> af, socktype, proto, canonname, sa = x[0]
>>> s = socket.socket(af, socktype, proto)
>>> s
<socket.socket fd=12, family=2, type=1, proto=6, laddr=('0.0.0.0', 0)>
>>> s.connect(sa)
>>> s.sendall(b"GET /\nHost: example.com\n\n")
>>> s.recv(100)
b'HTTP/1.0 200 OK\r\nAge: 352850\r\nCache-Control: max-age=604800\r\nContent-Type: text/html; charset=UTF-8\r'
>>> s.close()
```

If you are unable to resolve the hostname, or can't read/write to the port, then you probably need to fix the underlying cause in your own environment.
I tested this on 2023.1.0 and 2023.1.2 and saw no difference with my own services.
I ran the parameters you provided and I got this error

>>> s.connect(sa)
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/code.py", line 90, in runcode
    exec(code, self.locals)
  File "<console>", line 1, in <module>
ConnectionRefusedError: [Errno 111] Connection refused

I used your commands and my IP resolved. On the last command, I got an error 301 moved permanently instead of your response but other than that the IP resolved and the commands worked.
@jsaltzman Can you provide logs? You may not be having the same issue as OP.

@bohemtucsok What IP did the hostname resolve to? If it is a WAN address, authentik may be attempting to communicate LAN -> WAN -> LAN and is getting stopped by the firewall. Make sure you have the service domain name mapped to the correct LAN IP (or `127.0.0.1` if on the same machine) in your hosts file, Docker should correctly pass this to the container.
The ip is resolved well by nginx proxy manager localhost ip is internal, but you gave me an idea and I put 127.0.0.1 in the host file, and unfortunately the same error {"error":"Post \"https://auth.XXX. com/application/o/token/\": dial tcp 127.0.0.1:443: connect: connection refused","event":"


I'm clueless
127.0.0.1 in this case would reference to localhost within the container, so that shouldn't work in theory. The other thing that could cause this is if you're setting the docker hostname to the external FQDN in which case it will resolve to the authentik container itself and not the reverse proxy
the host name is not the same as the authentic fqdn
I meant setting `127.0.0.1` in the hosts file on the host machine. I run a local caching resolver (most *nix boxen do esp. with `systemd-resolved`) so [the Docker daemon's DNS forwards my `hosts` entries](https://docs.docker.com/config/containers/container-networking/#dns-services). In hindsight this might not apply to you or make sense in your environment. Instead try [`extra-hosts` in your `docker-compose.yml`](https://docs.docker.com/compose/compose-file/compose-file-v3/#extra_hosts) if you think this might help, but it sounds like that might not be the problem if the DNS is resolving as you expect.

What interface is the reverse proxy listening on? If it isn't listening on whatever interface the outpost is trying to talk to, then that could be causing this. If it's listening on all interfaces (`0.0.0.0`) this shouldn't be a problem.

@BeryJu I'm curious how the outposts were communicating to the main server before? Directly through the Docker interface without any lookups?
I don't know if I'm right, but I found an entry for proxy auth in 23.1.0

{"auth_via": "unauthenticated", "event": "/application/o/token/", "host": "localhost:8000",

as if the identification takes place on this port 8000, but at the same time in 23.1.2 it is not localhost but the proxy ip is 192.168.x.x.
@sevmonster I checked and the proxy listens on all interfaces
The situation is the following,

Until now, authentik ran on the same host as the nginx manager, I transferred the authentik docker to another host and since then the proxy in authentik works, I no longer get the error mentioned above.
So it probably was just the container unable to communicate with the LAN with however your network is configured. At least you got it working.
Hey guys

I managed to solve the problem :)
This applies to those who run the proxy manager and authenticators on the same host
The problem, as written above, is probably the docker LAN communication. For some reason, the packages go wrong on the interfaces, but this is just a tip :)

iptables solved the problem.
iptables -t nat -I PREROUTING --src 0/0 --dst 127.0.0.1 -p tcp --dport 80 -j REDIRECT --to-ports 80
iptables -t nat -I PREROUTING --src 0/0 --dst 127.0.0.1 -p tcp --dport 443 -j REDIRECT --to-ports 443
iptables -t nat -I PREROUTING --src 0/0 --dst 192.168.10.240 -p tcp --dport 443 -j REDIRECT --to-ports 443
iptables -t nat -I PREROUTING --src 0/0 --dst 192.168.10.240 -p tcp --dport 80 -j REDIRECT --to-ports 80

The point is that you redirect the traffic to the proxy manager's ports on the main network card and the localhost virtual interface.

I hope this will be a solution for many :)
Interesting, I wasn't aware this wasn't supported by default, do you have multiple interfdaces on your docker host?
the thing is very interesting, there is not only 1 network card in the vps, but due to docker there are several interfaces and this causes some confusion.
Right, as far as I knew this should work out of the box (reverse proxy and authentik on the same host, and authentik sending requests to itself via the reverse proxy), but I also haven't explicitly tested it
This worked fine until now, but the 23.1.2 update brought this error
yes because versions < 2023.1.0 did the OAuth token request internally, which was changed to using the external URL in 2023.1.0, as doing the call internally was causing the JWT token to have an invalid issuer
Great :) then the problem has been solved :)
@jsaltzman @SunRed Does the info above help?
> @jsaltzman @SunRed Does the info above help?

I resolved my issue. It was related to the fact that in the outpost I had my url set to http: and not https://
I had the same problem, but caused by a different mistake. I specified host and domain name in my docker compose file
```yaml
    hostname: authentik
    domainname: ${DOMAIN?err}
```
that caused the proxy look-up to give the docker internal ip for the authentik container. Now authentik does not listen on port 443, so the connection got refused. Removing the domainname line resulted in docker finding the correct ip and the request was routed through the reverse proxy.
Looks like I've got the same issue on my end. I tried the ak shell commands and received the same ConnectionRefusedError: [Errno 111] Connection refused after s.connect(sa). However, the iptables nat rules didn't seem to solve the issue for me. Hopefully there's a more concrete fix coming in a release soon.
Based on what others have said, the problem stems from host routing configuration. There does not appear to be anything that can be done in authentik, other than maybe adding an option to reintroduce local outpost communication, but that would also reintroduce the original bug that changing that process fixed. If you provide more information about your host setup, network configuration, and authentik configuration, someone might be able to help.
Sure. I've got NPM and Authentik running locally on my server. According to portainer the IP of NPM is 172.18.0.3. Per bohemtucsok's suggestions I've got the following iptables nat rules set on the host:

iptables -t nat -I PREROUTING --src 0/0 --dst 127.0.0.1 -p tcp --dport 80 -j REDIRECT --to-ports 80
iptables -t nat -I PREROUTING --src 0/0 --dst 127.0.0.1 -p tcp --dport 443 -j REDIRECT --to-ports 443
iptables -t nat -I PREROUTING --src 0/0 --dst 172.18.0.3 -p tcp --dport 80 -j REDIRECT --to-ports 80
iptables -t nat -I PREROUTING --src 0/0 --dst 172.18.0.3 -p tcp --dport 443 -j REDIRECT --to-ports 443

As far as network config, authentik's network is 172.19.0.0/16. My host is set statically at 192.168.200.75 behind a cable modem which receives a sticky static IP. 

From inside a console into the authentik server I'm able to ping NPM hosts but following the code example you provided earlier I also get hung up at s.connect(sa). I can provide any other information requested.
Rather than copy @bohemtucsok's configuration verbatim, it is better to understand what problem it is solving, because it is probably not equal to your situation. In most Docker configurations, the host network is isolated from the containers. There are some ways around this, including changing network type from bridged or whitelisting the host with `--add-host=host.docker.internal:host-gateway`, and you should probably try these before jumping to iptables.

The `REDIRECT` target basically DNATs the packet with the local interface IP. So for your modifications, any packet incoming to your local or loopback interface will be redirected to essentially `localhost:[to-port]`. This might be helpful in his configuration, but you mentioned Portainer, and I will ignorantly assume the network configuration with that tool is somehow different in that environment. You could also be specifying the wrong `to-port`, it should be whatever the proxy port is in your local environment. If you are behind a router that is doing port forwarding and the local port is not 80/443, well there's your problem.

If you want an `iptables` solution to work, you need to trace your own packets and see where they are going. Very rarely will anyone else's configuration work portably.

Resolving patch:
diff --git a/authentik/core/tasks.py b/authentik/core/tasks.py
--- a/authentik/core/tasks.py
+++ b/authentik/core/tasks.py
@@ -43,6 +43,7 @@ def clean_expired_models(self: MonitoredTask):
     amount = 0
     for session in AuthenticatedSession.objects.all():
         cache_key = f"{KEY_PREFIX}{session.session_key}"
+        value = None
         try:
             value = cache.get(cache_key)
         # pylint: disable=broad-except


Test patch:
diff --git a/authentik/policies/tests/test_engine.py b/authentik/policies/tests/test_engine.py
--- a/authentik/policies/tests/test_engine.py
+++ b/authentik/policies/tests/test_engine.py
@@ -5,6 +5,7 @@
 from authentik.core.models import User
 from authentik.policies.dummy.models import DummyPolicy
 from authentik.policies.engine import PolicyEngine
+from authentik.policies.exceptions import PolicyEngineException
 from authentik.policies.expression.models import ExpressionPolicy
 from authentik.policies.models import Policy, PolicyBinding, PolicyBindingModel, PolicyEngineMode
 from authentik.policies.tests.test_process import clear_policy_cache
@@ -93,7 +94,7 @@ def test_engine_policy_type(self):
         """Test invalid policy type"""
         pbm = PolicyBindingModel.objects.create()
         PolicyBinding.objects.create(target=pbm, policy=self.policy_wrong_type, order=0)
-        with self.assertRaises(TypeError):
+        with self.assertRaises(PolicyEngineException):
             engine = PolicyEngine(pbm, self.user)
             engine.build()