ofproto-dpif.at: Fix a race.

For the fixed line, the ofctl_monitor.log should have 18 lines.
So, should wait until it reaches 18 lines.

Signed-off-by: Alex Wang <alexw@nicira.com>
Acked-by: Joe Stringer <joestringer@nicira.com>
flow.h: Fix a comment typo

Acked-by: Jarno Rajahalme <jrajahalme@nicira.com>
Signed-off-by: YAMAMOTO Takashi <yamamoto@valinux.co.jp>
ofproto: New function ofproto_refresh_rule().

This function checks for a rule in the classifier:
* If the rule exists, reset its modified time.
* If an equivalent rule exists, reset that rule's modified time.
* If no rule exists, re-install the rule and reset its modified time.
* Finally, return the rule that was modified.

This function will be used to ensure that hard timeouts for learnt rules
are refreshed if traffic consistently hits a rule with a learn action in
it. The first user will be the next commit.

Signed-off-by: Joe Stringer <joestringer@nicira.com>
Acked-by: Ethan Jackson <ethan@nicira.com>
---
v2: Acked.
v1: Ensure rule->modified is updated correctly.
RFC: First post.
ofproto-dpif-xlate: Cache xlate_actions() effects.

This patch adds a new object called 'struct xlate_cache' which can be
set in 'struct xlate_in', and passed to xlate_actions() to cache the
modules affected by this flow translation. Subsequently, the caller can
pass the xcache to xlate_push_stats() to credit stats and perform side
effects for a lower cost than full flow translation.

These changes are aimed currently at long-lived flows, decreasing the
average dump duration for such flows by 50-80%. This allows more flows
to be supported in the datapath at a given time. Applying these changes
to short-lived flows is left for a later commit.

Signed-off-by: Joe Stringer <joestringer@nicira.com>
Acked-by: Ethan Jackson <ethan@nicira.com>
---
v2: Acked.
v1: Add caching for fin_timeout action.
    Expire netflows on xlate_cache_clear().
    Account to bonds using a copy of 'flow' rather than hash.
    Always build XC_NORMAL entry (previously only if may_learn is true)
    Rename xlate_from_cache()->xlate_push_stats()
    Add may_learn parameter to xlate_push_stats()
    Tidy up xlate_actions__() mirror/netflow code.
    Fold in style fixups.
RFC: First post.
revalidator: Only revalidate high-throughput flows.

Previously we would revalidate all flows if the "need_revalidate" flag
was raised. This patch modifies the logic to delete low throughput flows
rather than revalidate them. High-throughput flows are unaffected by
this change. This patch identifies the flows based on the mean time
between packets since the last dump.

This change is primarily targeted at situations where:
* Flow dump duration is high (~1 second)
* Revalidation is triggered. (eg, by bridge reconfiguration or learning)

After the need_revalidate flag is set, next time a new flow dump session
starts, revalidators will begin revalidating the flows. This full
revalidation is more expensive, which significantly increases the flow
dump duration. At the end of this dump session, the datapath flow
management algorithms kick in for the next dump:

* If flow dump duration becomes too long, the flow limit is decreased.
* The number of flows in the datapath then exceeds the flow_limit.
* As the flow_limit is exceeded, max_idle is temporarily set to 100ms.
* Revalidators delete all flows that haven't seen traffic recently.

The effect of this is that many low-throughput flows are deleted after
revalidation, even if they are valid. The revalidation is unnecessary
for flows that would be deleted anyway, so this patch skips the
revalidation step for those flows.

Note that this patch will only perform this optimization if the flow has
already been dumped at least once, and only if the time since the last
dump is sufficiently long. This gives the flow a chance to become
high-throughput.

Signed-off-by: Joe Stringer <joestringer@nicira.com>
Acked-by: Ethan Jackson <ethan@nicira.com>
---
v2: Acked.
v1: Determine "high-throughput" by packets rather than bytes.
    Calculate the mean time between packets for comparison, rather than
      comparing the number of packets since the last dump.
RFC: First post.
Merge pull request #3 from joestringer/submit/xlate_cache_v2

Cache the modules affected by xlate_actions().
lib/ofp-util: Restore the check for minus sign in port number strings.

Commit 33ab38d9 (meta-flow: Simplify mf_from_ofp_port_string())
inadvertently removed a check for minus sign at the beginning of a
port number string introduced by commit 05dddba (meta-flow: Don't
allow negative port numbers).  This check is still needed, so put it
back, but to ofputil_port_from_string() this time.

Signed-off-by: Jarno Rajahalme <jrajahalme@nicira.com>
Acked-by: Justin Pettit <jpettit@nicira.com>
Reviewed-by: YAMAMOTO Takashi <yamamoto@valinux.co.jp>
lib/flow: Introduce miniflow_extract().

miniflow_extract() extracts packet headers directly to a miniflow,
which is a compressed form of the struct flow.  This does not require
a large struct to be cleared to begin with, and accesses less memory.
These performance benefits should allow this to be used in the DPDK
datapath.

miniflow_extract() takes a miniflow as an input/output parameter.  On
input the buffer for values to be extracted must be properly
initialized.  On output the map contains ones for all the fields that
have been extracted.

Some struct flow fields are reordered to make miniflow_extract to
progress in the logical order.

Some explicit "inline" keywords are necessary for GCC to optimize this
properly.  Also, macros are used for same reason instead of inline
functions for pushing data to the miniflow.

Signed-off-by: Jarno Rajahalme <jrajahalme@nicira.com>
Reviewed-by: YAMAMOTO Takashi <yamamoto@valinux.co.jp>
lib/flow: Add miniflow accessors and miniflow_get_tcp_flags().

Add inlined generic accessors for miniflow integer type fields, and a
new miniflow_get_tcp_flags() usinge these.  These will be used in a
later patch.

Some definitions also used in lib/packets.h had to be moved there to
resolve circular include dependencies.  Similarly, some inline
functions using struct flow are now in lib/flow.h.  IMO this is
cleaner, since now the lib/flow.h need not be included from
lib/packets.h.

Signed-off-by: Jarno Rajahalme <jrajahalme@nicira.com>
Reviewed-by: YAMAMOTO Takashi <yamamoto@valinux.co.jp>
lib/flow: Possibly faster miniflow_hash_in_minimask()

Upcoming patches add classifier lookups using miniflows, this is
heavily used for it.

Signed-off-by: Jarno Rajahalme <jrajahalme@nicira.com>
Reviewed-by: YAMAMOTO Takashi <yamamoto@valinux.co.jp>
classifier: Support miniflow as a key.

Support struct miniflow as a key for datapath flow lookup.

The new classifier interface classifier_lookup_miniflow_first() takes
a miniflow as a key and stops at the first match with no regard to
flow prioritites.  This works only if the classifier has no
conflicting rules (as is the case with the userspace datapath
classifier).

Signed-off-by: Jarno Rajahalme <jrajahalme@nicira.com>
Reviewed-by: YAMAMOTO Takashi <yamamoto@valinux.co.jp>
dpif-netdev: Use miniflow as a flow key.

Use miniflow as a flow key in the userspace datapath classifier.  The
miniflow is expanded for upcalls, but for existing datapath flows, the
key need not be expanded.

Signed-off-by: Jarno Rajahalme <jrajahalme@nicira.com>
Reviewed-by: YAMAMOTO Takashi <yamamoto@valinux.co.jp>
dpif-linux: Implement the API functions to allow multiple handler threads read upcall.

Signed-off-by: Alex Wang <alexw@nicira.com>
Acked-by: Ethan Jackson <ethan@nicira.com>
dpif-linux: Pass 'struct dpif_linux *' to internal static functions.

This commit reformats the dpif-linux module so that all internal
static functions take 'struct dpif_linux *' as input argument.
This will allow the adding of thread-safety annotations.

Signed-off-by: Alex Wang <alexw@nicira.com>
Acked-by: Ethan Jackson <ethan@nicira.com>
dpif-linux: Add thread-safety annotations.

Signed-off-by: Alex Wang <alexw@nicira.com>
Acked-by: Ethan Jackson <ethan@nicira.com>
ofproto-dpif-upcall: Remove the dispatcher thread.

With the foundation laid in previous commits, this commit
removes the 'dispatcher' thread by allowing 'handler'
threads to read upcalls directly from dpif.

This commit significantly simplifies the flow miss handling
code and brings slight improvement to flow setup rate.

Signed-off-by: Alex Wang <alexw@nicira.com>
Acked-by: Ethan Jackson <ethan@nicira.com>
ofproto-dpif-upcall: Don't use stack garbage

Catched by "learning action - self-modifying flow with hard_timeout"
test case.

The bug introduced by commit b256dc52.
("ofproto-dpif-xlate: Cache xlate_actions() effects.")

Acked-by: Jarno Rajahalme <jrajahalme@nicira.com>
Signed-off-by: YAMAMOTO Takashi <yamamoto@valinux.co.jp>
ofproto-macros.at: Assert daemon commands creates pidfile before exiting

I observed that "ofproto-dpif - ofproto-dpif-monitor 2" test,
which merely accesses ovsdb, failed in OVS_VSWITCHD_STOP's
"ovs-appctl -t ovs-vswitchd exit".  This commit tries to catch
the problem earlier by sprinkling AT_CHECKs.
ofproto-dpif.at: Fix some races

These races were exposed on my environment by commit c2a77f33.
("tests/ofproto-dpif: Use vlog to test dpif behaviour.")
ofproto-dpif.at: Use OVS_APP_EXIT_AND_WAIT where appropriate

Leftover of commit 0c473314.
("ofproto-dpif.at: Wait for the monitor's pidfile disappears where necessary")