Q:

How do I do the bash equivalent of $PROGPATH/program in Powershell? 

In GNU/Linux I would do:
PROGPATH=/long/and/complicated/path/to/some/bin
$PROGPATH/program args...

but in Powershell if I try this:
$PROGPATH=\long\and\complicated\path\to\some\bin
$PROGPATH\program args...

I get:
At script.ps1:2 char:...
+ $PROGPATH\program args ...
+          ~~~~~~~~
Unexpected token '\program' in expression or statement. 
+ CategoryInfo          : ParserError: (:) [], ParseException
+ FullyQualifiedErrorId : UnexpectedToken

So how do I do this simple thing I know how to do in bash, in Powershell? 

A:

js2010's helpful answer shows the correct solution: 
Because your command name/path starts with a variable reference ($PROGPATH/...), you must invoke it with &. 
Note that the same applies if a command name/path is quoted ('...' or "...")[1], as is required if the path contains spaces, for instance. 
As for why:
&, the call operator is necessary to force interpretation of a statement as a command, i.e. to have it parsed in argument mode (see below), so as to result in command execution rather than expression evaluation. 
PowerShell has two fundamental parsing modes:

argument mode, which works like a traditional shell, where the first token is a command name/path, such as a cmdlet or an external program, with subsequent tokens representing the arguments, which only require quoting if they contain shell metacharacters (chars.  with special meaning to PowerShell, such as spaces to separate tokens). 
expression mode, which works like expressions in programming languages. 

PowerShell decides based on a statement's first token what parsing mode to apply:
If, among other things, the first token starts with a variable reference or is a quoted string, PowerShell parses in expression mode. 

In expression mode, \ starts a new token, and unrecognized token \program results in the syntax error you saw. 
(If you had used /, it would have been interpreted as the division operator, and program wouldn't be a valid divisor operand.) 

[1] Note that if your executable path is a string literal (doesn't contain variable references) you may alternatively `-escape individual characters (spaces) in lieu of enclosing entire string in '...' or "...", in which case & is then not necessary; e.g.:
C:\Program` Files\Notepad++\notepad++.exe

